[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:05:36 dist-test-slave-hgs0 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:05:36 dist-test-slave-hgs0 krb5kdc[395](info): set up 2 sockets
Sep 27 05:05:36 dist-test-slave-hgs0 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 05:05:37.673403 388 mini_kdc.cc:121] Time spent starting KDC: real 2.198s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:05:37 dist-test-slave-hgs0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:05:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:05:37.866073 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37291
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:38.235792 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:38.236153 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:38.236321 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:38.245067 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:05:38.245222 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:38.245345 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:38.245450 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:05:38.245548 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:05:38.257076 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:38.258620 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:38.260264 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:38.275614 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:38.275843 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:38.276401 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:38.276763 408 server_base.cc:1034] running on GCE node
I20260927 05:05:38.277814 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:38.279557 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:38.280828 408 hybrid_clock.cc:648] HybridClock initialized: now 1790485538280693 us; error 157 us; skew 500 ppm
I20260927 05:05:38.281280 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:38 dist-test-slave-hgs0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:38.289887 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:05:38.291795 408 webserver.cc:460] Webserver started at http://127.0.97.62:41147/ using document root <none> and password file <none>
I20260927 05:05:38.292765 408 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:38.292887 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:38.293196 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:38.296861 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "86e2e7a4e2b04c28877ce399c64b7471"
format_stamp: "Formatted at 2026-09-27 05:05:38 on dist-test-slave-hgs0"
server_key: "8e6726b6c9f8c487b2836b50313a7b36"
server_key_iv: "fbe93214d0e0778be726444368d7a9d8"
server_key_version: "encryptionkey@0"
I20260927 05:05:38.297765 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "86e2e7a4e2b04c28877ce399c64b7471"
format_stamp: "Formatted at 2026-09-27 05:05:38 on dist-test-slave-hgs0"
server_key: "8e6726b6c9f8c487b2836b50313a7b36"
server_key_iv: "fbe93214d0e0778be726444368d7a9d8"
server_key_version: "encryptionkey@0"
I20260927 05:05:38.303777 408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 05:05:38.308056 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:38.309669 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:05:38.309846 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "86e2e7a4e2b04c28877ce399c64b7471"
format_stamp: "Formatted at 2026-09-27 05:05:38 on dist-test-slave-hgs0"
server_key: "8e6726b6c9f8c487b2836b50313a7b36"
server_key_iv: "fbe93214d0e0778be726444368d7a9d8"
server_key_version: "encryptionkey@0"
I20260927 05:05:38.310093 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:38.344347 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:38.350577 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:38.350965 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:38.378649 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37291
I20260927 05:05:38.378690 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37291 every 8 connection(s)
I20260927 05:05:38.380975 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:05:38.386698 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:38.388262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 408
I20260927 05:05:38.388625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:05:38.391772 388 external_mini_cluster.cc:1428] Setting key a44d0c9ce3d2eead98a9417a1b10511c
I20260927 05:05:38.407907 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471: Bootstrap starting.
I20260927 05:05:38.412549 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:38.414448 475 log.cc:824] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:38.419442 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471: No bootstrap required, opened a new log
I20260927 05:05:38.430145 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e2e7a4e2b04c28877ce399c64b7471" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37291 } }
Sep 27 05:05:38 dist-test-slave-hgs0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:38.431147 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:38.431293 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e2e7a4e2b04c28877ce399c64b7471, State: Initialized, Role: FOLLOWER
I20260927 05:05:38.432019 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e2e7a4e2b04c28877ce399c64b7471" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37291 } }
I20260927 05:05:38.432330 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:05:38.432487 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:05:38.432688 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:38.435498 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e2e7a4e2b04c28877ce399c64b7471" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37291 } }
I20260927 05:05:38.436197 475 leader_election.cc:302] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86e2e7a4e2b04c28877ce399c64b7471; no voters:
I20260927 05:05:38.436920 475 leader_election.cc:288] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:05:38.437129 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:38.439414 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:05:38.440129 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [term 1 LEADER]: Becoming Leader. State: Replica: 86e2e7a4e2b04c28877ce399c64b7471, State: Running, Role: LEADER
I20260927 05:05:38.440809 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e2e7a4e2b04c28877ce399c64b7471" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37291 } }
I20260927 05:05:38.444772 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86e2e7a4e2b04c28877ce399c64b7471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e2e7a4e2b04c28877ce399c64b7471" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37291 } } }
I20260927 05:05:38.445240 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [sys.catalog]: This master's current role is: LEADER
I20260927 05:05:38.446403 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:38.399056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.97.62:37291)
0927 05:05:38.400045 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:38.400062 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:38.400972 (+ 910us) server_negotiation.cc:407] Connection header received
0927 05:05:38.412830 (+ 11858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:38.412864 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:38.413508 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:38.419066 (+ 5558us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:38.419706 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:38.421374 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:38.424164 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:38.424915 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:38.425159 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:38.433057 (+ 7898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:38.433067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:38.433085 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:38.433137 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:38.437310 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:38.439303 (+ 1993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:38.439318 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:38.439332 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:38.439682 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:38.440429 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:38.440435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:38.440439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:38.441735 (+ 1296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:38.442061 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:38.444057 (+ 1996us) server_negotiation.cc:299] Negotiation successful
0927 05:05:38.444605 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":431,"threads_started":1}
I20260927 05:05:38.447610 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86e2e7a4e2b04c28877ce399c64b7471. Latest consensus state: current_term: 1 leader_uuid: "86e2e7a4e2b04c28877ce399c64b7471" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e2e7a4e2b04c28877ce399c64b7471" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37291 } } }
I20260927 05:05:38.447916 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471 [sys.catalog]: This master's current role is: LEADER
I20260927 05:05:38.452412 488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:05:38.458827 488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:05:38.470528 488 catalog_manager.cc:1347] Generated new cluster ID: 429199b49bf34d9c9c99573310a82dd4
I20260927 05:05:38.470685 488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:05:38.506217 488 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:05:38.508455 488 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:05:38.521195 488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86e2e7a4e2b04c28877ce399c64b7471: Generated new TSK 0
I20260927 05:05:38.522487 488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:05:38.576399 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37291
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:38.968777 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:38.969117 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:38.969236 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:38.978114 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:38.978259 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:38.978402 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:05:38.989122 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37291
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:38.990470 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:38.991798 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:39.006783 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:39.006965 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:39.006801 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:39.008970 501 server_base.cc:1034] running on GCE node
I20260927 05:05:39.009781 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:39.011294 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:39.012537 501 hybrid_clock.cc:648] HybridClock initialized: now 1790485539012491 us; error 50 us; skew 500 ppm
I20260927 05:05:39.013022 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:39 dist-test-slave-hgs0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:39.017256 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:05:39.018918 501 webserver.cc:460] Webserver started at http://127.0.97.1:35705/ using document root <none> and password file <none>
I20260927 05:05:39.019730 501 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:39.019863 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:39.020292 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:39.023496 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "109eaa52e9354e53a30093d9aae5f16a"
format_stamp: "Formatted at 2026-09-27 05:05:39 on dist-test-slave-hgs0"
server_key: "cc08f1645e05c9fe6132d318ad09bd3a"
server_key_iv: "55b1a6a27590ffa4115d02fb670bdf13"
server_key_version: "encryptionkey@0"
I20260927 05:05:39.024457 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "109eaa52e9354e53a30093d9aae5f16a"
format_stamp: "Formatted at 2026-09-27 05:05:39 on dist-test-slave-hgs0"
server_key: "cc08f1645e05c9fe6132d318ad09bd3a"
server_key_iv: "55b1a6a27590ffa4115d02fb670bdf13"
server_key_version: "encryptionkey@0"
I20260927 05:05:39.030611 501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:05:39.035297 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:39.037061 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:05:39.037245 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "109eaa52e9354e53a30093d9aae5f16a"
format_stamp: "Formatted at 2026-09-27 05:05:39 on dist-test-slave-hgs0"
server_key: "cc08f1645e05c9fe6132d318ad09bd3a"
server_key_iv: "55b1a6a27590ffa4115d02fb670bdf13"
server_key_version: "encryptionkey@0"
I20260927 05:05:39.037456 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:39.072376 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:39.076793 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:39.077102 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:39.078073 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:05:39.079430 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:39.079504 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:39.079622 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:39.079684 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:39.119084 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34531
I20260927 05:05:39.119130 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34531 every 8 connection(s)
I20260927 05:05:39.120527 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:05:39.127506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 501
I20260927 05:05:39.127774 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:05:39.128518 388 external_mini_cluster.cc:1428] Setting key e622db4e742fe3d44b18f93287239710
Sep 27 05:05:39 dist-test-slave-hgs0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:39.144122 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:39.123336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52823 (local address 127.0.97.62:37291)
0927 05:05:39.123841 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:39.123847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:39.125116 (+ 1269us) server_negotiation.cc:407] Connection header received
0927 05:05:39.127221 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:39.127228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:39.127340 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:39.127480 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:39.130156 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:39.131320 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:39.133185 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:39.133623 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:39.133782 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:39.137812 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:39.137818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:39.137821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:39.137856 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:39.140652 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:39.141588 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:39.141600 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:39.141603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:39.141730 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:39.142213 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:39.142217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:39.142219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:39.142567 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:39.142735 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:39.143487 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 05:05:39.143714 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
I20260927 05:05:39.145176 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:39.124075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37291 (local address 127.0.97.1:52823)
0927 05:05:39.124898 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:39.124953 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:39.126809 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:39.127828 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:39.127839 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:39.128553 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:39.129899 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:39.129932 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:39.131488 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:39.131507 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:39.133007 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:39.133022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:39.133896 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:39.133906 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:39.134167 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:39.134931 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:39.134956 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:39.137532 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:39.140814 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:39.140824 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:39.140844 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:39.141446 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:39.141857 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:39.141862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:39.141866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:39.142093 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:39.142700 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:39.142708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:39.143085 (+ 377us) client_negotiation.cc:769] Sending connection context
0927 05:05:39.143380 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 05:05:39.143983 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":346,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:05:39.146960 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37291
I20260927 05:05:39.147279 629 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:05:39.148276 629 heartbeater.cc:511] Master 127.0.97.62:37291 requested a full tablet report, sending...
I20260927 05:05:39.150915 440 ts_manager.cc:194] Registered new tserver with Master: 109eaa52e9354e53a30093d9aae5f16a (127.0.97.1:34531)
I20260927 05:05:39.153524 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:05:39.172870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37291
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:39.537829 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:39.538161 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:39.538344 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:39.546176 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:39.546306 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:39.546530 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:05:39.557211 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37291
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:39.558573 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:39.560056 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:39.573613 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:39.574259 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:39.574265 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:39.575212 635 server_base.cc:1034] running on GCE node
I20260927 05:05:39.575987 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:39.576977 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:39.578214 635 hybrid_clock.cc:648] HybridClock initialized: now 1790485539578149 us; error 60 us; skew 500 ppm
I20260927 05:05:39.578552 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:39 dist-test-slave-hgs0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:39.581929 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:05:39.583511 635 webserver.cc:460] Webserver started at http://127.0.97.2:40825/ using document root <none> and password file <none>
I20260927 05:05:39.584326 635 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:39.584478 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:39.584797 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:39.587906 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "9944ba5aa5a44279bb3b57d6caa3d98c"
format_stamp: "Formatted at 2026-09-27 05:05:39 on dist-test-slave-hgs0"
server_key: "a60e41d5a1033e9085ef5b588936212f"
server_key_iv: "82121bf64d26385395574432d4851da8"
server_key_version: "encryptionkey@0"
I20260927 05:05:39.588840 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9944ba5aa5a44279bb3b57d6caa3d98c"
format_stamp: "Formatted at 2026-09-27 05:05:39 on dist-test-slave-hgs0"
server_key: "a60e41d5a1033e9085ef5b588936212f"
server_key_iv: "82121bf64d26385395574432d4851da8"
server_key_version: "encryptionkey@0"
I20260927 05:05:39.594663 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 05:05:39.599126 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:39.601389 635 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 05:05:39.601653 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "9944ba5aa5a44279bb3b57d6caa3d98c"
format_stamp: "Formatted at 2026-09-27 05:05:39 on dist-test-slave-hgs0"
server_key: "a60e41d5a1033e9085ef5b588936212f"
server_key_iv: "82121bf64d26385395574432d4851da8"
server_key_version: "encryptionkey@0"
I20260927 05:05:39.601948 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:39.624992 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:39.630112 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:39.630487 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:39.631628 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:05:39.633134 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:39.633245 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:39.633406 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:39.633489 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:39.674252 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35785
I20260927 05:05:39.674320 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35785 every 8 connection(s)
I20260927 05:05:39.676314 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:05:39.682229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 635
I20260927 05:05:39.682463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:05:39.682950 388 external_mini_cluster.cc:1428] Setting key 8c246bff8b2914baafc57172a31c0b05
Sep 27 05:05:39 dist-test-slave-hgs0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:05:39.702800 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:39.679402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57779 (local address 127.0.97.62:37291)
0927 05:05:39.679898 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:39.679904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:39.681227 (+ 1323us) server_negotiation.cc:407] Connection header received
0927 05:05:39.683342 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:39.683348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:39.683465 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:39.683615 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:39.685739 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:39.686490 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:39.689699 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:39.690452 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:39.690615 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:39.694950 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:39.694960 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:39.694968 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:39.695017 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:39.699383 (+ 4366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:39.700351 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:39.700357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:39.700362 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:39.700524 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:39.700953 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:39.700958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:39.700961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:39.701378 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:39.701539 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:39.702284 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 05:05:39.702454 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":225,"threads_started":1}
I20260927 05:05:39.703914 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:39.680165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37291 (local address 127.0.97.2:57779)
0927 05:05:39.681037 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:39.681085 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:39.682985 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:39.683948 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:39.683993 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:39.684609 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:39.685506 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:39.685536 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:39.686674 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:39.686682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:39.689496 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:39.689517 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:39.690763 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:39.690772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:39.691156 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:39.692083 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:39.692114 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:39.694646 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:39.699587 (+ 4941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:39.699595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:39.699612 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:39.700198 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:39.700643 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:39.700646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:39.700648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:39.700825 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:39.701506 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:39.701512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:39.701930 (+ 418us) client_negotiation.cc:769] Sending connection context
0927 05:05:39.702166 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 05:05:39.702591 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":358,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:05:39.705616 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37291
I20260927 05:05:39.705978 763 heartbeater.cc:464] Registering TS with master...
I20260927 05:05:39.706967 763 heartbeater.cc:511] Master 127.0.97.62:37291 requested a full tablet report, sending...
I20260927 05:05:39.709220 440 ts_manager.cc:194] Registered new tserver with Master: 9944ba5aa5a44279bb3b57d6caa3d98c (127.0.97.2:35785)
I20260927 05:05:39.710470 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:05:39.728018 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37291
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:40.083438 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:40.083770 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:40.083912 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:40.091914 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:40.092072 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:40.092286 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:05:40.103570 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37291
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:40.105265 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:40.106873 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:40.125849 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:40.126204 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:40.126497 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:40.126937 769 server_base.cc:1034] running on GCE node
I20260927 05:05:40.127633 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:40.128746 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:40.130012 769 hybrid_clock.cc:648] HybridClock initialized: now 1790485540129927 us; error 89 us; skew 500 ppm
I20260927 05:05:40.130385 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:40.133970 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:05:40.135730 769 webserver.cc:460] Webserver started at http://127.0.97.3:36083/ using document root <none> and password file <none>
I20260927 05:05:40.136569 769 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:40.136722 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:40.137072 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:40.140509 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "e3525acaa2b24a05bc4a29fac13a17fc"
format_stamp: "Formatted at 2026-09-27 05:05:40 on dist-test-slave-hgs0"
server_key: "e64857390fe087a311ea54d7259e6b3c"
server_key_iv: "5d08beb1626b167935fca09ffc80e212"
server_key_version: "encryptionkey@0"
I20260927 05:05:40.141451 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e3525acaa2b24a05bc4a29fac13a17fc"
format_stamp: "Formatted at 2026-09-27 05:05:40 on dist-test-slave-hgs0"
server_key: "e64857390fe087a311ea54d7259e6b3c"
server_key_iv: "5d08beb1626b167935fca09ffc80e212"
server_key_version: "encryptionkey@0"
I20260927 05:05:40.147029 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:05:40.151355 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:40.152894 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:05:40.153110 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "e3525acaa2b24a05bc4a29fac13a17fc"
format_stamp: "Formatted at 2026-09-27 05:05:40 on dist-test-slave-hgs0"
server_key: "e64857390fe087a311ea54d7259e6b3c"
server_key_iv: "5d08beb1626b167935fca09ffc80e212"
server_key_version: "encryptionkey@0"
I20260927 05:05:40.153379 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:40.157011 629 heartbeater.cc:503] Master 127.0.97.62:37291 was elected leader, sending a full tablet report...
I20260927 05:05:40.168519 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:40.172875 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:40.173197 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:40.174189 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:05:40.175606 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:40.175714 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:40.175866 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:40.175973 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:40.215900 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42741
I20260927 05:05:40.215909 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42741 every 8 connection(s)
I20260927 05:05:40.217294 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:05:40.227062 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 769
I20260927 05:05:40.227308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:05:40.227988 388 external_mini_cluster.cc:1428] Setting key cc627d1325caad893bc07efd0fb44116
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:40.242199 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:40.219906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41269 (local address 127.0.97.62:37291)
0927 05:05:40.220435 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:40.220439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:40.221416 (+ 977us) server_negotiation.cc:407] Connection header received
0927 05:05:40.223346 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:40.223352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:40.223460 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:40.223597 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:40.225365 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.226171 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.228888 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.229568 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.229706 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:40.234925 (+ 5219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:40.234933 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:40.234938 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:40.234995 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:40.238372 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.239357 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.239362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.239365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.239598 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.240203 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.240208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.240210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.240636 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:40.240776 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:40.241708 (+ 932us) server_negotiation.cc:299] Negotiation successful
0927 05:05:40.241915 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":231,"threads_started":1}
I20260927 05:05:40.243299 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:40.220504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37291 (local address 127.0.97.3:41269)
0927 05:05:40.221228 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:40.221284 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:40.223029 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:40.223866 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:40.223876 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:40.224507 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:40.225160 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:40.225181 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.226345 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.226358 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:40.228642 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:40.228664 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.229942 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.229955 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:40.230319 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:40.231322 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:40.231367 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:40.234596 (+ 3229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:40.238590 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:40.238600 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:40.238618 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:40.239142 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:40.239733 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:40.239738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:40.239742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:40.240042 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:40.240798 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:40.240808 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:40.241218 (+ 410us) client_negotiation.cc:769] Sending connection context
0927 05:05:40.241513 (+ 295us) client_negotiation.cc:240] Negotiation successful
0927 05:05:40.242083 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":282,"threads_started":1}
I20260927 05:05:40.244971 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37291
I20260927 05:05:40.245345 897 heartbeater.cc:464] Registering TS with master...
I20260927 05:05:40.246212 897 heartbeater.cc:511] Master 127.0.97.62:37291 requested a full tablet report, sending...
I20260927 05:05:40.247860 439 ts_manager.cc:194] Registered new tserver with Master: e3525acaa2b24a05bc4a29fac13a17fc (127.0.97.3:42741)
I20260927 05:05:40.248916 439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41269
I20260927 05:05:40.258879 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485540, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485540, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:05:40.310528 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:40.292491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.0.97.1:34531)
0927 05:05:40.293448 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:40.293460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:40.293664 (+ 204us) server_negotiation.cc:407] Connection header received
0927 05:05:40.293899 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:40.293907 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:40.294214 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:40.294408 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:40.296021 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.297126 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.298552 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.299238 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.299360 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:40.303910 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:40.303926 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:40.304026 (+ 100us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:40.304079 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:40.307476 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.308361 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.308370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.308373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.308534 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.308934 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.308937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.308940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.309322 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:40.309576 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:40.309855 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 05:05:40.310088 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":469,"threads_started":1}
W20260927 05:05:40.311880 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46210
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:05:40.360219 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:40.344162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.0.97.1:34531)
0927 05:05:40.344423 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:40.344433 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:40.344813 (+ 380us) server_negotiation.cc:407] Connection header received
0927 05:05:40.345070 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:40.345074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:40.345190 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:40.345313 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:40.346833 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.347641 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.349167 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.349640 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.349792 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:40.354399 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:40.354405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:40.354408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:40.354452 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:40.357459 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.358186 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.358191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.358193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.358324 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.358740 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.358745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.358749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.359033 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:40.359286 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:40.359623 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 05:05:40.359855 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:05:40.362661 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 501
I20260927 05:05:40.377641 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:40.363698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46220 (local address 127.0.97.1:34531)
0927 05:05:40.363935 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:40.363943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:40.364312 (+ 369us) server_negotiation.cc:407] Connection header received
0927 05:05:40.364522 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:40.364527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:40.364589 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:40.364703 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:40.366004 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.366799 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.368353 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.368840 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.368997 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:40.371503 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:40.371509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:40.371513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:40.371552 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:40.374725 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.375489 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.375496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.375500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.375653 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.376097 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.376103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.376106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.376389 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:40.376662 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:40.376997 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 05:05:40.377247 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:05:40.378436 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:05:40.471597 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 501
I20260927 05:05:40.489419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 635
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:05:40.507120 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:40.490525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.0.97.2:35785)
0927 05:05:40.491176 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:40.491184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:40.491211 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:05:40.491275 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:40.491282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:40.491553 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:40.491696 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:40.493046 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.494162 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.495644 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.496375 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.496491 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:40.500808 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:40.500823 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:40.500840 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:40.500891 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:40.503886 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.504857 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.504867 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.504869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.505044 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.505455 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.505459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.505461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.505840 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:40.506099 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:40.506472 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 05:05:40.506729 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":337,"threads_started":1}
I20260927 05:05:40.508781 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:05:40.597568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 635
I20260927 05:05:40.620590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 769
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:05:40.638655 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:40.621875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.0.97.3:42741)
0927 05:05:40.622592 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:40.622600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:40.622633 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:05:40.622704 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:40.622711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:40.622951 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:40.623109 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:40.624604 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.625789 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.627319 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:40.628167 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:40.628308 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:40.633064 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:40.633080 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:40.633098 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:40.633147 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:40.636098 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.636803 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.636811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.636813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.636936 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:40.637263 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:40.637266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:40.637268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:40.637600 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:40.637820 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:40.638158 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 05:05:40.638362 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":372,"threads_started":1}
I20260927 05:05:40.640192 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:05:40.729487 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 769
I20260927 05:05:40.746784 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 408
I20260927 05:05:40.748183 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:05:40.860110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 408
2026-09-27T05:05:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5441 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[919](info): set up 2 sockets
Sep 27 05:05:40 dist-test-slave-hgs0 krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260927 05:05:41.988396 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:05:42 dist-test-slave-hgs0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:05:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:05:42.180876 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45375
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:42.525053 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:42.525470 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:42.525646 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:42.533209 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:05:42.533390 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:42.533505 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:42.533629 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:05:42.533722 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:05:42.545758 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:42.547416 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:42.548900 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:42.562016 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:42.562140 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:42.562287 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:42.563544 932 server_base.cc:1034] running on GCE node
I20260927 05:05:42.564561 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:42.566283 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:42.567556 932 hybrid_clock.cc:648] HybridClock initialized: now 1790485542567504 us; error 44 us; skew 500 ppm
I20260927 05:05:42.568054 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:42 dist-test-slave-hgs0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:42.577601 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:05:42.579326 932 webserver.cc:460] Webserver started at http://127.0.97.62:35685/ using document root <none> and password file <none>
I20260927 05:05:42.580222 932 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:42.580375 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:42.580718 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:42.584048 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "3342c40c7e8246b5be79391ed5317111"
format_stamp: "Formatted at 2026-09-27 05:05:42 on dist-test-slave-hgs0"
server_key: "ea7c16b6a4b94d196f25db0e6ff5f940"
server_key_iv: "ccbac80684329dab7513b6315f784c82"
server_key_version: "encryptionkey@0"
I20260927 05:05:42.584913 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "3342c40c7e8246b5be79391ed5317111"
format_stamp: "Formatted at 2026-09-27 05:05:42 on dist-test-slave-hgs0"
server_key: "ea7c16b6a4b94d196f25db0e6ff5f940"
server_key_iv: "ccbac80684329dab7513b6315f784c82"
server_key_version: "encryptionkey@0"
I20260927 05:05:42.590615 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 05:05:42.594606 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:42.596170 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:05:42.596410 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "3342c40c7e8246b5be79391ed5317111"
format_stamp: "Formatted at 2026-09-27 05:05:42 on dist-test-slave-hgs0"
server_key: "ea7c16b6a4b94d196f25db0e6ff5f940"
server_key_iv: "ccbac80684329dab7513b6315f784c82"
server_key_version: "encryptionkey@0"
I20260927 05:05:42.596661 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:42.620102 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:42.625324 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:42.625702 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:42.644830 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45375
I20260927 05:05:42.644853 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45375 every 8 connection(s)
I20260927 05:05:42.646457 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:05:42.649777 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 932
I20260927 05:05:42.650032 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:05:42.650636 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:42.650771 388 external_mini_cluster.cc:1428] Setting key c0563c9c8e936733450ff12445dfd36a
I20260927 05:05:42.666857 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111: Bootstrap starting.
Sep 27 05:05:42 dist-test-slave-hgs0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:42.670312 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:42.671440 999 log.cc:824] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:42.674539 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111: No bootstrap required, opened a new log
I20260927 05:05:42.676904 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:42.653043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.0.97.62:45375)
0927 05:05:42.654007 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:42.654024 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:42.654071 (+ 47us) server_negotiation.cc:407] Connection header received
0927 05:05:42.656024 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:42.656052 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:42.656532 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:42.656938 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:42.658949 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:42.660414 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:42.662229 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:42.663039 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:42.663201 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:42.668369 (+ 5168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:42.668381 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:42.668402 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:42.668464 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:42.672119 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:42.672968 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:42.672977 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:42.672985 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:42.673174 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:42.673627 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:42.673634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:42.673637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:42.674220 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:42.674575 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:42.675186 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 05:05:42.675644 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":442,"threads_started":1}
I20260927 05:05:42.683288 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3342c40c7e8246b5be79391ed5317111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45375 } }
I20260927 05:05:42.683866 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:42.684090 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3342c40c7e8246b5be79391ed5317111, State: Initialized, Role: FOLLOWER
I20260927 05:05:42.684871 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3342c40c7e8246b5be79391ed5317111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45375 } }
I20260927 05:05:42.685245 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:05:42.685424 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:05:42.685621 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:42.689018 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3342c40c7e8246b5be79391ed5317111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45375 } }
I20260927 05:05:42.689692 999 leader_election.cc:302] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3342c40c7e8246b5be79391ed5317111; no voters:
I20260927 05:05:42.690429 999 leader_election.cc:288] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:05:42.690553 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:42.692286 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [term 1 LEADER]: Becoming Leader. State: Replica: 3342c40c7e8246b5be79391ed5317111, State: Running, Role: LEADER
I20260927 05:05:42.692986 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3342c40c7e8246b5be79391ed5317111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45375 } }
I20260927 05:05:42.693608 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:05:42.696018 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3342c40c7e8246b5be79391ed5317111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3342c40c7e8246b5be79391ed5317111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45375 } } }
I20260927 05:05:42.696427 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [sys.catalog]: This master's current role is: LEADER
I20260927 05:05:42.698424 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3342c40c7e8246b5be79391ed5317111. Latest consensus state: current_term: 1 leader_uuid: "3342c40c7e8246b5be79391ed5317111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3342c40c7e8246b5be79391ed5317111" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45375 } } }
I20260927 05:05:42.698853 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111 [sys.catalog]: This master's current role is: LEADER
I20260927 05:05:42.700686 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:05:42.712016 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:05:42.721915 1009 catalog_manager.cc:1347] Generated new cluster ID: a4259ebf1c4940cc9ba35bfb70750f1d
I20260927 05:05:42.722076 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:05:42.733533 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:05:42.735292 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:05:42.742498 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3342c40c7e8246b5be79391ed5317111: Generated new TSK 0
I20260927 05:05:42.743676 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:05:42.807824 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45375
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:43.179840 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:43.180230 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:43.180418 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:43.187919 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:43.188165 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:43.188351 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:05:43.199241 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:43.201021 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:43.202749 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:43.217352 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:43.217442 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:43.217842 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:43.218680 1025 server_base.cc:1034] running on GCE node
I20260927 05:05:43.219404 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:43.220571 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:43.221830 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790485543221748 us; error 67 us; skew 500 ppm
I20260927 05:05:43.222184 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:43 dist-test-slave-hgs0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:43.225701 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:05:43.227286 1025 webserver.cc:460] Webserver started at http://127.0.97.1:36625/ using document root <none> and password file <none>
I20260927 05:05:43.228181 1025 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:43.228371 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:43.228753 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:43.231995 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "77812fb3aadf4bfc8769ce9af7177b9e"
format_stamp: "Formatted at 2026-09-27 05:05:43 on dist-test-slave-hgs0"
server_key: "d2fe9cce85dad78a79956004246047eb"
server_key_iv: "069f7aaad46dddaef030706c8cc0147d"
server_key_version: "encryptionkey@0"
I20260927 05:05:43.232940 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "77812fb3aadf4bfc8769ce9af7177b9e"
format_stamp: "Formatted at 2026-09-27 05:05:43 on dist-test-slave-hgs0"
server_key: "d2fe9cce85dad78a79956004246047eb"
server_key_iv: "069f7aaad46dddaef030706c8cc0147d"
server_key_version: "encryptionkey@0"
I20260927 05:05:43.239295 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 05:05:43.243386 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:43.244956 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:05:43.245164 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "77812fb3aadf4bfc8769ce9af7177b9e"
format_stamp: "Formatted at 2026-09-27 05:05:43 on dist-test-slave-hgs0"
server_key: "d2fe9cce85dad78a79956004246047eb"
server_key_iv: "069f7aaad46dddaef030706c8cc0147d"
server_key_version: "encryptionkey@0"
I20260927 05:05:43.245384 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:43.270026 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:43.274559 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:43.274871 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:43.275862 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:05:43.277236 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:43.277340 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:43.277494 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:43.277566 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:43.311316 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46751
I20260927 05:05:43.311368 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46751 every 8 connection(s)
I20260927 05:05:43.312740 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:05:43.320123 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 1025
I20260927 05:05:43.320340 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:05:43.320940 388 external_mini_cluster.cc:1428] Setting key f8d4b6e4aff0fda053bf4a2e0e4a6dc1
Sep 27 05:05:43 dist-test-slave-hgs0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:43.334712 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:43.315463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43679 (local address 127.0.97.62:45375)
0927 05:05:43.315919 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:43.315924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:43.316812 (+ 888us) server_negotiation.cc:407] Connection header received
0927 05:05:43.318665 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:43.318670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:43.318766 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:43.318885 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:43.320639 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:43.321416 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:43.323577 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:43.324082 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:43.324188 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:43.328675 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:43.328681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:43.328684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:43.328719 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:43.331300 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:43.332192 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:43.332203 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:43.332206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:43.332327 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:43.332805 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:43.332810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:43.332813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:43.333099 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:43.333257 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:43.334105 (+ 848us) server_negotiation.cc:299] Negotiation successful
0927 05:05:43.334346 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":236,"threads_started":1}
I20260927 05:05:43.335846 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:43.315860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45375 (local address 127.0.97.1:43679)
0927 05:05:43.316638 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:43.316685 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:43.318385 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:43.319138 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:43.319147 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:43.319684 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:43.320459 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:43.320492 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:43.321622 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:43.321626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:43.323404 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:43.323422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:43.324322 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:43.324333 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:43.324697 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:43.325525 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:43.325563 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:43.328405 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:43.331477 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:43.331487 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:43.331507 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:43.332056 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:43.332449 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:43.332454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:43.332467 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:43.332673 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:43.333241 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:43.333249 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:43.333709 (+ 460us) client_negotiation.cc:769] Sending connection context
0927 05:05:43.333987 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 05:05:43.334579 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":263,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:05:43.337693 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45375
I20260927 05:05:43.338047 1153 heartbeater.cc:464] Registering TS with master...
I20260927 05:05:43.338979 1153 heartbeater.cc:511] Master 127.0.97.62:45375 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:05:43.341655 964 ts_manager.cc:194] Registered new tserver with Master: 77812fb3aadf4bfc8769ce9af7177b9e (127.0.97.1:46751)
I20260927 05:05:43.343273 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:05:43.364567 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45375
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:43.708232 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:43.708570 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:43.708781 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:43.717176 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:43.717319 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:43.717532 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:05:43.728009 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:43.729560 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:43.731151 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:43.746219 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:43.746166 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:43.746860 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:43.747225 1159 server_base.cc:1034] running on GCE node
I20260927 05:05:43.748077 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:43.749377 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:43.750619 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790485543750562 us; error 97 us; skew 500 ppm
I20260927 05:05:43.751066 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:43 dist-test-slave-hgs0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:43.754817 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:05:43.756705 1159 webserver.cc:460] Webserver started at http://127.0.97.2:39135/ using document root <none> and password file <none>
I20260927 05:05:43.757536 1159 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:43.757714 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:43.758083 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:43.761600 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "e15abb783ee341a5a8d180573480a468"
format_stamp: "Formatted at 2026-09-27 05:05:43 on dist-test-slave-hgs0"
server_key: "8c426d8f5787cd58ac3de8bc405bbeb5"
server_key_iv: "f13d2e79e50f3d083d58f637e9e92c33"
server_key_version: "encryptionkey@0"
I20260927 05:05:43.762545 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e15abb783ee341a5a8d180573480a468"
format_stamp: "Formatted at 2026-09-27 05:05:43 on dist-test-slave-hgs0"
server_key: "8c426d8f5787cd58ac3de8bc405bbeb5"
server_key_iv: "f13d2e79e50f3d083d58f637e9e92c33"
server_key_version: "encryptionkey@0"
I20260927 05:05:43.769141 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 05:05:43.773702 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:43.775482 1159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:05:43.775785 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "e15abb783ee341a5a8d180573480a468"
format_stamp: "Formatted at 2026-09-27 05:05:43 on dist-test-slave-hgs0"
server_key: "8c426d8f5787cd58ac3de8bc405bbeb5"
server_key_iv: "f13d2e79e50f3d083d58f637e9e92c33"
server_key_version: "encryptionkey@0"
I20260927 05:05:43.776117 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:43.802577 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:43.807351 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:43.807621 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:43.808676 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:05:43.810039 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:43.810119 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:43.810209 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:43.810253 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:43.844449 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39331
I20260927 05:05:43.844553 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39331 every 8 connection(s)
I20260927 05:05:43.845819 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:05:43.854172 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 1159
I20260927 05:05:43.854379 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:05:43.854874 388 external_mini_cluster.cc:1428] Setting key a66847a57dade7728617c2966a71949f
Sep 27 05:05:43 dist-test-slave-hgs0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:43.869266 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:43.848523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33541 (local address 127.0.97.62:45375)
0927 05:05:43.849080 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:43.849085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:43.849977 (+ 892us) server_negotiation.cc:407] Connection header received
0927 05:05:43.851835 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:43.851840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:43.851931 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:43.852068 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:43.853611 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:43.854796 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:43.856782 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:43.857315 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:43.857420 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:43.862068 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:43.862073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:43.862077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:43.862113 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:43.865166 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:43.866341 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:43.866346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:43.866349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:43.866539 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:43.867130 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:43.867135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:43.867137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:43.867481 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:43.867638 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:43.868681 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0927 05:05:43.868965 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":263,"threads_started":1}
I20260927 05:05:43.871001 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:43.849092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45375 (local address 127.0.97.2:33541)
0927 05:05:43.849803 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:43.849858 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:43.851547 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:43.852313 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:43.852322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:43.852837 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:43.853423 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:43.853444 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:43.855002 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:43.855016 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:43.856620 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:43.856642 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:43.857577 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:43.857586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:43.857964 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:43.858782 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:43.858823 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:43.861758 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:43.865360 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:43.865373 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:43.865398 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:43.866180 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:43.866679 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:43.866687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:43.866694 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:43.866976 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:43.867638 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:43.867650 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:43.868230 (+ 580us) client_negotiation.cc:769] Sending connection context
0927 05:05:43.868590 (+ 360us) client_negotiation.cc:240] Negotiation successful
0927 05:05:43.869270 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":267,"threads_started":1}
I20260927 05:05:43.872848 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45375
I20260927 05:05:43.873236 1287 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:05:43.874131 1287 heartbeater.cc:511] Master 127.0.97.62:45375 requested a full tablet report, sending...
I20260927 05:05:43.875988 964 ts_manager.cc:194] Registered new tserver with Master: e15abb783ee341a5a8d180573480a468 (127.0.97.2:39331)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:05:43.876909 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:05:43.899500 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45375
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:44.256083 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:44.256460 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:44.256655 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:44.264803 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:44.265110 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:44.265383 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:05:44.277567 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:44.279136 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:44.280804 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:44.295995 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:44.296167 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:44.296864 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:44.297596 1293 server_base.cc:1034] running on GCE node
I20260927 05:05:44.298359 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:44.299772 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:44.301048 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790485544300951 us; error 97 us; skew 500 ppm
I20260927 05:05:44.301524 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:44 dist-test-slave-hgs0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:44.305738 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:05:44.307686 1293 webserver.cc:460] Webserver started at http://127.0.97.3:44545/ using document root <none> and password file <none>
I20260927 05:05:44.308598 1293 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:44.308779 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:44.309157 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:44.312618 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "81f8be9ab2824e899a057f8315ad6edf"
format_stamp: "Formatted at 2026-09-27 05:05:44 on dist-test-slave-hgs0"
server_key: "ce5845c3603e28a365ad1371ff9814a2"
server_key_iv: "170403a6514aa6acb9d9eee1065c0c00"
server_key_version: "encryptionkey@0"
I20260927 05:05:44.313534 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "81f8be9ab2824e899a057f8315ad6edf"
format_stamp: "Formatted at 2026-09-27 05:05:44 on dist-test-slave-hgs0"
server_key: "ce5845c3603e28a365ad1371ff9814a2"
server_key_iv: "170403a6514aa6acb9d9eee1065c0c00"
server_key_version: "encryptionkey@0"
I20260927 05:05:44.319780 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 05:05:44.324123 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:44.325738 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:05:44.325953 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "81f8be9ab2824e899a057f8315ad6edf"
format_stamp: "Formatted at 2026-09-27 05:05:44 on dist-test-slave-hgs0"
server_key: "ce5845c3603e28a365ad1371ff9814a2"
server_key_iv: "170403a6514aa6acb9d9eee1065c0c00"
server_key_version: "encryptionkey@0"
I20260927 05:05:44.326180 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:44.347244 1153 heartbeater.cc:503] Master 127.0.97.62:45375 was elected leader, sending a full tablet report...
I20260927 05:05:44.356276 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:44.360646 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:44.360888 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:44.361842 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:05:44.363184 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:44.363257 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:44.363340 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:44.363382 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:44.400199 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41227
I20260927 05:05:44.400243 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41227 every 8 connection(s)
I20260927 05:05:44.401733 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:05:44.409189 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 1293
I20260927 05:05:44.409420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:05:44.409873 388 external_mini_cluster.cc:1428] Setting key e4726fe94a1402894f87395bd5b23e88
Sep 27 05:05:44 dist-test-slave-hgs0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:44.423478 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.404673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37913 (local address 127.0.97.62:45375)
0927 05:05:44.405174 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:44.405180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:44.406334 (+ 1154us) server_negotiation.cc:407] Connection header received
0927 05:05:44.408527 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:44.408533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:44.408643 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:44.408794 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:44.410711 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.411800 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.413894 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.414309 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.414409 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.417835 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:44.417840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:44.417843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:44.417876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:44.420831 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:44.421533 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:44.421537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:44.421539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:44.421652 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:44.422022 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:44.422025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:44.422027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:44.422269 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:44.422373 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:44.423032 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 05:05:44.423193 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":222,"threads_started":1}
I20260927 05:05:44.424842 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.405356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45375 (local address 127.0.97.3:37913)
0927 05:05:44.406133 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:44.406188 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:44.408199 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:44.409072 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:44.409082 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:44.409773 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:44.410477 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.410508 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.412039 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.412045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.413763 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.413775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.414523 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.414528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.414731 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.415379 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:44.415400 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:44.417628 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:44.420970 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:44.420977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:44.420993 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:44.421412 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:44.421739 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:44.421743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:44.421745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:44.421917 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:44.422367 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:44.422373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:44.422702 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 05:05:44.422923 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 05:05:44.423310 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":300,"threads_started":1}
I20260927 05:05:44.426168 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45375
I20260927 05:05:44.426481 1421 heartbeater.cc:464] Registering TS with master...
I20260927 05:05:44.427204 1421 heartbeater.cc:511] Master 127.0.97.62:45375 requested a full tablet report, sending...
I20260927 05:05:44.428676 964 ts_manager.cc:194] Registered new tserver with Master: 81f8be9ab2824e899a057f8315ad6edf (127.0.97.3:41227)
I20260927 05:05:44.429503 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37913
I20260927 05:05:44.438297 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:05:44.460150 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.444814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.0.97.62:45375)
0927 05:05:44.445047 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:44.445056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:44.445481 (+ 425us) server_negotiation.cc:407] Connection header received
0927 05:05:44.445735 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:44.445739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:44.445812 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:44.445929 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:44.447537 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.448442 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.450540 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.451015 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.451179 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.454524 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:44.454530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:44.454534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:44.454577 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:44.457187 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:44.457934 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:44.457940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:44.457944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:44.458089 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:44.458509 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:44.458516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:44.458520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:44.458873 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:44.459150 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:44.459505 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 05:05:44.459758 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:05:44.467103 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:05:44.474511 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:05:44.502753 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.489258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41227 (local address 127.0.0.1:50850)
0927 05:05:44.490002 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:44.490061 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:44.490283 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:44.492244 (+ 1961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:44.492253 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:44.492300 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:44.492707 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.492721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.494857 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.494864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.498509 (+ 3645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.498529 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.501423 (+ 2894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.501431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.501750 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.501835 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 05:05:44.502003 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 05:05:44.502252 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":900,"threads_started":1}
I20260927 05:05:44.505028 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.489281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39331 (local address 127.0.0.1:58892)
0927 05:05:44.493350 (+ 4069us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:44.493372 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:44.493498 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:44.494526 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:44.494532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:44.494560 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:44.495182 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.495196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.498178 (+ 2982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.498185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.501126 (+ 2941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.501184 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.504053 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.504060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.504284 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.504341 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 05:05:44.504449 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 05:05:44.504603 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3942,"thread_start_us":3798,"threads_started":1}
I20260927 05:05:44.506012 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.489332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.0.97.2:39331)
0927 05:05:44.492941 (+ 3609us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:44.492950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:44.493473 (+ 523us) server_negotiation.cc:407] Connection header received
0927 05:05:44.493678 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:44.493686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:44.494001 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:44.494199 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:44.495893 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.497961 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.501445 (+ 3484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.503445 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.503612 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.503875 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:44.505458 (+ 1583us) server_negotiation.cc:299] Negotiation successful
0927 05:05:44.505669 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3414,"thread_start_us":3277,"threads_started":1}
I20260927 05:05:44.506193 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.489871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.97.3:41227)
0927 05:05:44.491524 (+ 1653us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:44.491533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:44.491566 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:05:44.491671 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:44.491680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:44.492011 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:44.492236 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:44.492954 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.494673 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.498819 (+ 4146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.500894 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.501081 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.505530 (+ 4449us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:44.505620 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 05:05:44.505815 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452,"thread_start_us":1764,"threads_started":1}
I20260927 05:05:44.510465 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 126c5fe481a4414b92f17990ce8ef548 (DEFAULT_TABLE table=test-table [id=48dc78670ed34afdaf936ff554fd6340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:44.510514 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 126c5fe481a4414b92f17990ce8ef548 (DEFAULT_TABLE table=test-table [id=48dc78670ed34afdaf936ff554fd6340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:44.512146 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126c5fe481a4414b92f17990ce8ef548. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:44.512152 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126c5fe481a4414b92f17990ce8ef548. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:44.514604 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.495655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46751 (local address 127.0.0.1:38648)
0927 05:05:44.503855 (+ 8200us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:44.503877 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:44.504059 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:44.507346 (+ 3287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:44.507351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:44.507385 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:44.507592 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.507604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.509637 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.509643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.511518 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.511536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.513663 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.513670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.513875 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.513912 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:05:44.514046 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 05:05:44.514238 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8077,"thread_start_us":10152,"threads_started":1}
I20260927 05:05:44.515077 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.493982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.0.97.1:46751)
0927 05:05:44.506552 (+ 12570us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:44.506559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:44.506589 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:05:44.506729 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:44.506739 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:44.507093 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:44.507345 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:44.507772 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.509420 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.511739 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.513407 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.513592 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.513858 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:44.514122 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 05:05:44.514744 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12394,"thread_start_us":12251,"threads_started":1}
I20260927 05:05:44.519385 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 126c5fe481a4414b92f17990ce8ef548 (DEFAULT_TABLE table=test-table [id=48dc78670ed34afdaf936ff554fd6340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:44.521036 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126c5fe481a4414b92f17990ce8ef548. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:44.530208 1441 tablet_bootstrap.cc:492] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468: Bootstrap starting.
I20260927 05:05:44.530385 1440 tablet_bootstrap.cc:492] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf: Bootstrap starting.
I20260927 05:05:44.534226 1440 tablet_bootstrap.cc:654] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:44.534667 1441 tablet_bootstrap.cc:654] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:44.535629 1440 log.cc:824] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:44.536010 1441 log.cc:824] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:44.538355 1444 tablet_bootstrap.cc:492] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e: Bootstrap starting.
I20260927 05:05:44.539649 1441 tablet_bootstrap.cc:492] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468: No bootstrap required, opened a new log
I20260927 05:05:44.539628 1440 tablet_bootstrap.cc:492] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf: No bootstrap required, opened a new log
I20260927 05:05:44.539983 1440 ts_tablet_manager.cc:1397] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 05:05:44.540000 1441 ts_tablet_manager.cc:1397] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260927 05:05:44.542963 1444 tablet_bootstrap.cc:654] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:44.544459 1444 log.cc:824] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:44.548906 1444 tablet_bootstrap.cc:492] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e: No bootstrap required, opened a new log
I20260927 05:05:44.549204 1440 raft_consensus.cc:348] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15abb783ee341a5a8d180573480a468" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39331 } } peers { permanent_uuid: "81f8be9ab2824e899a057f8315ad6edf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41227 } } peers { permanent_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46751 } }
I20260927 05:05:44.549644 1440 raft_consensus.cc:374] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:44.549798 1444 ts_tablet_manager.cc:1397] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260927 05:05:44.549852 1440 raft_consensus.cc:729] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f8be9ab2824e899a057f8315ad6edf, State: Initialized, Role: FOLLOWER
I20260927 05:05:44.550709 1440 consensus_queue.cc:260] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15abb783ee341a5a8d180573480a468" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39331 } } peers { permanent_uuid: "81f8be9ab2824e899a057f8315ad6edf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41227 } } peers { permanent_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46751 } }
I20260927 05:05:44.552083 1421 heartbeater.cc:503] Master 127.0.97.62:45375 was elected leader, sending a full tablet report...
I20260927 05:05:44.552858 1440 ts_tablet_manager.cc:1428] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260927 05:05:44.552613 1441 raft_consensus.cc:348] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15abb783ee341a5a8d180573480a468" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39331 } } peers { permanent_uuid: "81f8be9ab2824e899a057f8315ad6edf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41227 } } peers { permanent_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46751 } }
I20260927 05:05:44.553164 1441 raft_consensus.cc:374] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:44.553284 1441 raft_consensus.cc:729] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15abb783ee341a5a8d180573480a468, State: Initialized, Role: FOLLOWER
I20260927 05:05:44.554164 1441 consensus_queue.cc:260] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15abb783ee341a5a8d180573480a468" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39331 } } peers { permanent_uuid: "81f8be9ab2824e899a057f8315ad6edf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41227 } } peers { permanent_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46751 } }
I20260927 05:05:44.555496 1287 heartbeater.cc:503] Master 127.0.97.62:45375 was elected leader, sending a full tablet report...
I20260927 05:05:44.557130 1441 ts_tablet_manager.cc:1428] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260927 05:05:44.561690 1444 raft_consensus.cc:348] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15abb783ee341a5a8d180573480a468" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39331 } } peers { permanent_uuid: "81f8be9ab2824e899a057f8315ad6edf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41227 } } peers { permanent_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46751 } }
I20260927 05:05:44.562083 1444 raft_consensus.cc:374] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:44.562217 1444 raft_consensus.cc:729] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77812fb3aadf4bfc8769ce9af7177b9e, State: Initialized, Role: FOLLOWER
I20260927 05:05:44.562806 1444 consensus_queue.cc:260] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15abb783ee341a5a8d180573480a468" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39331 } } peers { permanent_uuid: "81f8be9ab2824e899a057f8315ad6edf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41227 } } peers { permanent_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46751 } }
I20260927 05:05:44.564651 1444 ts_tablet_manager.cc:1428] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
W20260927 05:05:44.565335 1154 tablet.cc:2369] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:05:44.598067 1288 tablet.cc:2369] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:05:44.653532 1422 tablet.cc:2369] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:05:44.961025 1446 raft_consensus.cc:482] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:05:44.961318 1446 raft_consensus.cc:504] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15abb783ee341a5a8d180573480a468" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39331 } } peers { permanent_uuid: "81f8be9ab2824e899a057f8315ad6edf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41227 } } peers { permanent_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46751 } }
I20260927 05:05:44.962872 1446 leader_election.cc:288] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e15abb783ee341a5a8d180573480a468 (127.0.97.2:39331), 77812fb3aadf4bfc8769ce9af7177b9e (127.0.97.1:46751)
I20260927 05:05:44.970769 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.963342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52925 (local address 127.0.97.2:39331)
0927 05:05:44.963532 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:44.963536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:44.963971 (+ 435us) server_negotiation.cc:407] Connection header received
0927 05:05:44.964234 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:44.964244 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:44.964340 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:44.964404 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:44.965025 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.966010 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.968523 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.969758 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.969912 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.970025 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:44.970298 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 05:05:44.970399 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:05:44.970845 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.963287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39331 (local address 127.0.97.3:52925)
0927 05:05:44.963833 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:44.963858 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:44.964039 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:44.964593 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:44.964596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:44.964616 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:44.964871 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.964879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.966142 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.966146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.968345 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.968360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.969936 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.969940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.970095 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.970143 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 05:05:44.970247 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:05:44.970397 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":3,"thread_start_us":320,"threads_started":1}
I20260927 05:05:44.971001 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.964039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46751 (local address 127.0.97.3:36727)
0927 05:05:44.964553 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:44.964572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:44.964745 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:44.965098 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:44.965101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:44.965119 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:44.965386 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.965393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.966596 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.966601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.968611 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:44.968620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.970193 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.970201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:44.970379 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.970406 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 05:05:44.970500 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 05:05:44.970631 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":52,"thread_start_us":246,"threads_started":1}
I20260927 05:05:44.971043 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:44.964324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36727 (local address 127.0.97.1:46751)
0927 05:05:44.964551 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:44.964557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:44.964669 (+ 112us) server_negotiation.cc:407] Connection header received
0927 05:05:44.964886 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:44.964892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:44.964983 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:44.965078 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:44.965563 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.966452 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.968748 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:44.969986 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:44.970119 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:44.970188 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:44.970645 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 05:05:44.970777 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:05:44.972091 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126c5fe481a4414b92f17990ce8ef548" candidate_uuid: "81f8be9ab2824e899a057f8315ad6edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" is_pre_election: true
I20260927 05:05:44.972579 1108 raft_consensus.cc:2454] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f8be9ab2824e899a057f8315ad6edf in term 0.
I20260927 05:05:44.972163 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126c5fe481a4414b92f17990ce8ef548" candidate_uuid: "81f8be9ab2824e899a057f8315ad6edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e15abb783ee341a5a8d180573480a468" is_pre_election: true
I20260927 05:05:44.973169 1242 raft_consensus.cc:2454] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81f8be9ab2824e899a057f8315ad6edf in term 0.
I20260927 05:05:44.973241 1310 leader_election.cc:302] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77812fb3aadf4bfc8769ce9af7177b9e, 81f8be9ab2824e899a057f8315ad6edf; no voters:
I20260927 05:05:44.973703 1446 raft_consensus.cc:2781] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:05:44.973863 1446 raft_consensus.cc:482] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:05:44.973976 1446 raft_consensus.cc:3037] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:44.976413 1446 raft_consensus.cc:504] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15abb783ee341a5a8d180573480a468" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39331 } } peers { permanent_uuid: "81f8be9ab2824e899a057f8315ad6edf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41227 } } peers { permanent_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46751 } }
I20260927 05:05:44.977023 1446 leader_election.cc:288] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [CANDIDATE]: Term 1 election: Requested vote from peers e15abb783ee341a5a8d180573480a468 (127.0.97.2:39331), 77812fb3aadf4bfc8769ce9af7177b9e (127.0.97.1:46751)
I20260927 05:05:44.977639 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126c5fe481a4414b92f17990ce8ef548" candidate_uuid: "81f8be9ab2824e899a057f8315ad6edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e15abb783ee341a5a8d180573480a468"
I20260927 05:05:44.977669 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126c5fe481a4414b92f17990ce8ef548" candidate_uuid: "81f8be9ab2824e899a057f8315ad6edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77812fb3aadf4bfc8769ce9af7177b9e"
I20260927 05:05:44.977857 1242 raft_consensus.cc:3037] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:44.977873 1108 raft_consensus.cc:3037] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:44.980247 1108 raft_consensus.cc:2454] T 126c5fe481a4414b92f17990ce8ef548 P 77812fb3aadf4bfc8769ce9af7177b9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f8be9ab2824e899a057f8315ad6edf in term 1.
I20260927 05:05:44.980274 1242 raft_consensus.cc:2454] T 126c5fe481a4414b92f17990ce8ef548 P e15abb783ee341a5a8d180573480a468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81f8be9ab2824e899a057f8315ad6edf in term 1.
I20260927 05:05:44.980841 1310 leader_election.cc:302] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77812fb3aadf4bfc8769ce9af7177b9e, 81f8be9ab2824e899a057f8315ad6edf; no voters:
I20260927 05:05:44.981196 1446 raft_consensus.cc:2781] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:44.981827 1446 raft_consensus.cc:686] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [term 1 LEADER]: Becoming Leader. State: Replica: 81f8be9ab2824e899a057f8315ad6edf, State: Running, Role: LEADER
I20260927 05:05:44.982355 1446 consensus_queue.cc:237] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15abb783ee341a5a8d180573480a468" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39331 } } peers { permanent_uuid: "81f8be9ab2824e899a057f8315ad6edf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41227 } } peers { permanent_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46751 } }
I20260927 05:05:44.987534 962 catalog_manager.cc:5495] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf reported cstate change: term changed from 0 to 1, leader changed from <none> to 81f8be9ab2824e899a057f8315ad6edf (127.0.97.3). New cstate: current_term: 1 leader_uuid: "81f8be9ab2824e899a057f8315ad6edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e15abb783ee341a5a8d180573480a468" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81f8be9ab2824e899a057f8315ad6edf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77812fb3aadf4bfc8769ce9af7177b9e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46751 } health_report { overall_health: UNKNOWN } } }
Sep 27 05:05:45 dist-test-slave-hgs0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:05:45.080129 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:45.063551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.0.97.1:46751)
0927 05:05:45.063786 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:45.063795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:45.064279 (+ 484us) server_negotiation.cc:407] Connection header received
0927 05:05:45.064543 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:45.064548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:45.064618 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:45.064734 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:45.066246 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.067210 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.069161 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.069620 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.069782 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:45.074119 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:45.074137 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:45.074154 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:45.074196 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:45.077128 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.077830 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.077839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.077842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.077984 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.078422 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.078427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.078429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.078806 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:45.079131 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:45.079454 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 05:05:45.079713 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 05:05:45 dist-test-slave-hgs0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485545, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 05:05:45 dist-test-slave-hgs0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485545, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:05:45.131006 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:45.114892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.0.97.1:46751)
0927 05:05:45.115204 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:45.115216 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:45.115602 (+ 386us) server_negotiation.cc:407] Connection header received
0927 05:05:45.115884 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:45.115890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:45.116037 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:45.116169 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:45.117860 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.118710 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.120274 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.120751 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.120908 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:45.125681 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:45.125687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:45.125691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:45.125732 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:45.128363 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.129093 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.129099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.129101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.129240 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.129668 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.129674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.129687 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.129975 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:45.130166 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:45.130503 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 05:05:45.130699 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
Sep 27 05:05:45 dist-test-slave-hgs0 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 05:05:45 dist-test-slave-hgs0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:05:45.180148 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:45.165117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.0.97.1:46751)
0927 05:05:45.165428 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:45.165435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:45.165687 (+ 252us) server_negotiation.cc:407] Connection header received
0927 05:05:45.165842 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:45.165846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:45.165915 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:45.166010 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:45.167432 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.168367 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.169659 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.170127 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.170239 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:45.174272 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:45.174278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:45.174282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:45.174323 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:45.177231 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.178010 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.178015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.178017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.178158 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.178530 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.178535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.178538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.178856 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:45.179102 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:45.179332 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 05:05:45.179604 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260927 05:05:45.182117 1068 tablet_service.cc:2908] Rejecting scan request for tablet 126c5fe481a4414b92f17990ce8ef548: Uninitialized: clean time has not yet been initialized
I20260927 05:05:45.185782 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 1025
I20260927 05:05:45.200204 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:45.186839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.0.97.1:46751)
0927 05:05:45.187015 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:45.187020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:45.187450 (+ 430us) server_negotiation.cc:407] Connection header received
0927 05:05:45.187603 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:45.187607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:45.187700 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:45.187806 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:45.189645 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.190353 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.191667 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.192173 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.192275 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:45.194811 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:45.194817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:45.194821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:45.194866 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:45.197564 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.198286 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.198290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.198292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.198417 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.198743 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.198746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.198748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.199025 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:45.199248 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:45.199624 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 05:05:45.199819 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:05:45.201041 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:05:45.291930 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 1025
I20260927 05:05:45.310231 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 1159
Sep 27 05:05:45 dist-test-slave-hgs0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:05:45.326877 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:45.311041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.0.97.2:39331)
0927 05:05:45.311239 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:45.311245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:45.311266 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:05:45.311449 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:45.311455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:45.311537 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:45.311662 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:45.313055 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.313784 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.315145 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.315587 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.315737 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:45.320108 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:45.320126 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:45.320145 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:45.320186 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:45.323469 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.324466 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.324477 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.324480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.324636 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.325107 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.325112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.325115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.325548 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:45.325912 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:45.326243 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 05:05:45.326504 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:05:45.327674 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:05:45.382731 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:45.382137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46751 (local address 127.0.97.3:54647)
0927 05:05:45.382386 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:45.382550 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 05:05:45.383035 1310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46751: connect: Connection refused (error 111)
I20260927 05:05:45.384739 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:45.384314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46751 (local address 127.0.97.3:45339)
0927 05:05:45.384483 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:45.384569 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:05:45.384953 1310 consensus_peers.cc:487] T 126c5fe481a4414b92f17990ce8ef548 P 81f8be9ab2824e899a057f8315ad6edf -> Peer 77812fb3aadf4bfc8769ce9af7177b9e (127.0.97.1:46751): Couldn't send request to peer 77812fb3aadf4bfc8769ce9af7177b9e. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 05:05:45.422050 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 1159
I20260927 05:05:45.440117 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 1293
Sep 27 05:05:45 dist-test-slave-hgs0 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:05:45.456498 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:45.440902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.0.97.3:41227)
0927 05:05:45.441418 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:45.441422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:45.441441 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:05:45.441500 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:45.441503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:45.441560 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:45.441689 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:45.443029 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.443895 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.445285 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:45.445687 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:45.445824 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:45.450166 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:45.450184 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:45.450203 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:45.450243 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:45.453465 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.454198 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.454207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.454209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.454362 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:45.454828 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:45.454832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:45.454834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:45.455253 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:45.455486 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:45.455819 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 05:05:45.456167 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":246,"threads_started":1}
I20260927 05:05:45.457213 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:05:45.556041 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 1293
I20260927 05:05:45.572954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 932
I20260927 05:05:45.574177 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:05:45.685873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 932
2026-09-27T05:05:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4827 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:05:45 dist-test-slave-hgs0 krb5kdc[1475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:05:45 dist-test-slave-hgs0 krb5kdc[1475](info): set up 2 sockets
Sep 27 05:05:45 dist-test-slave-hgs0 krb5kdc[1475](info): commencing operation
krb5kdc: starting...
W20260927 05:05:46.804040 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:05:46 dist-test-slave-hgs0 krb5kdc[1475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:05:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:05:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:05:46.995330 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32845
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:47.341113 1488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:47.341424 1488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:47.341614 1488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:47.349116 1488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:05:47.349284 1488 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 05:05:47.349370 1488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:47.349476 1488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:47.349565 1488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:05:47.349650 1488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:05:47.360013 1488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32845
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:47.361438 1488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:47.362807 1488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:47.374423 1493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:47.374523 1496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:47.374553 1494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:47.375707 1488 server_base.cc:1034] running on GCE node
I20260927 05:05:47.376541 1488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:47.377955 1488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:47.379184 1488 hybrid_clock.cc:648] HybridClock initialized: now 1790485547379116 us; error 39 us; skew 500 ppm
I20260927 05:05:47.379529 1488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:47 dist-test-slave-hgs0 krb5kdc[1475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:47.382752 1488 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:05:47.384418 1488 webserver.cc:460] Webserver started at http://127.0.97.62:38277/ using document root <none> and password file <none>
I20260927 05:05:47.385174 1488 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:47.385308 1488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:47.385641 1488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:47.388803 1488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "149fef5995694f15a8349f4ce4fc219a"
format_stamp: "Formatted at 2026-09-27 05:05:47 on dist-test-slave-hgs0"
server_key: "d6c922d4eb335056f05cf6ae44aee211"
server_key_iv: "e31735f5aa4d8bf14f8ac1a793d84a92"
server_key_version: "encryptionkey@0"
I20260927 05:05:47.389678 1488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "149fef5995694f15a8349f4ce4fc219a"
format_stamp: "Formatted at 2026-09-27 05:05:47 on dist-test-slave-hgs0"
server_key: "d6c922d4eb335056f05cf6ae44aee211"
server_key_iv: "e31735f5aa4d8bf14f8ac1a793d84a92"
server_key_version: "encryptionkey@0"
I20260927 05:05:47.395051 1488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:05:47.398829 1503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:47.400492 1488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:05:47.400699 1488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "149fef5995694f15a8349f4ce4fc219a"
format_stamp: "Formatted at 2026-09-27 05:05:47 on dist-test-slave-hgs0"
server_key: "d6c922d4eb335056f05cf6ae44aee211"
server_key_iv: "e31735f5aa4d8bf14f8ac1a793d84a92"
server_key_version: "encryptionkey@0"
I20260927 05:05:47.400946 1488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:47.425484 1488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:47.430244 1488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:47.430570 1488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:47.454264 1488 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32845
I20260927 05:05:47.454362 1564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32845 every 8 connection(s)
I20260927 05:05:47.455821 1488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:05:47.460485 1565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:47.464927 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 1488
I20260927 05:05:47.465232 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:05:47.465902 388 external_mini_cluster.cc:1428] Setting key fce308fec1197a7cda76dc846e84c83b
I20260927 05:05:47.474915 1565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a: Bootstrap starting.
I20260927 05:05:47.478541 1565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:47.479696 1565 log.cc:824] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:05:47 dist-test-slave-hgs0 krb5kdc[1475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:47.482921 1565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a: No bootstrap required, opened a new log
I20260927 05:05:47.490557 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:47.468365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.0.97.62:32845)
0927 05:05:47.469259 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:47.469273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:47.469321 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:05:47.471107 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:47.471136 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:47.471639 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:47.472177 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:47.473749 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:47.475238 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:47.476699 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:47.477446 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:47.477670 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:47.482157 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:47.482166 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:47.482183 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:47.482226 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:47.485648 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:47.486668 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:47.486676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:47.486683 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:47.486882 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:47.487332 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:47.487336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:47.487338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:47.488020 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:47.488270 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:47.488833 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 05:05:47.489346 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":403,"threads_started":1}
I20260927 05:05:47.492659 1565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fef5995694f15a8349f4ce4fc219a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32845 } }
I20260927 05:05:47.493079 1565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:47.493221 1565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149fef5995694f15a8349f4ce4fc219a, State: Initialized, Role: FOLLOWER
I20260927 05:05:47.493832 1565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fef5995694f15a8349f4ce4fc219a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32845 } }
I20260927 05:05:47.494117 1565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:05:47.494261 1565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:05:47.494432 1565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:47.496713 1565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fef5995694f15a8349f4ce4fc219a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32845 } }
I20260927 05:05:47.497243 1565 leader_election.cc:302] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 149fef5995694f15a8349f4ce4fc219a; no voters:
I20260927 05:05:47.497889 1565 leader_election.cc:288] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:05:47.498041 1570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:47.498975 1570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [term 1 LEADER]: Becoming Leader. State: Replica: 149fef5995694f15a8349f4ce4fc219a, State: Running, Role: LEADER
I20260927 05:05:47.499577 1570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fef5995694f15a8349f4ce4fc219a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32845 } }
I20260927 05:05:47.500186 1565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:05:47.502681 1571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "149fef5995694f15a8349f4ce4fc219a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fef5995694f15a8349f4ce4fc219a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32845 } } }
I20260927 05:05:47.503072 1571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [sys.catalog]: This master's current role is: LEADER
I20260927 05:05:47.503649 1572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 149fef5995694f15a8349f4ce4fc219a. Latest consensus state: current_term: 1 leader_uuid: "149fef5995694f15a8349f4ce4fc219a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "149fef5995694f15a8349f4ce4fc219a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32845 } } }
I20260927 05:05:47.503925 1572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a [sys.catalog]: This master's current role is: LEADER
I20260927 05:05:47.504060 1578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:05:47.510648 1578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:05:47.522692 1578 catalog_manager.cc:1347] Generated new cluster ID: 483365efabbc4cd98ea6a640fb2a0fcb
I20260927 05:05:47.522856 1578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:05:47.535386 1578 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:05:47.537174 1578 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:05:47.546020 1578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a: Generated new TSK 0
I20260927 05:05:47.547046 1578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:05:47.609766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32845
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:47.965196 1591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:47.965549 1591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:47.965720 1591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:47.972999 1591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:47.973161 1591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:47.973323 1591 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:05:47.973475 1591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:05:47.983485 1591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:32845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:47.984897 1591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:47.986346 1591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:48.000474 1599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:48.000435 1596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:48.000381 1597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:48.001284 1591 server_base.cc:1034] running on GCE node
I20260927 05:05:48.002010 1591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:48.002920 1591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:48.004141 1591 hybrid_clock.cc:648] HybridClock initialized: now 1790485548004083 us; error 36 us; skew 500 ppm
I20260927 05:05:48.004478 1591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:48 dist-test-slave-hgs0 krb5kdc[1475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:48.013391 1591 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:05:48.014882 1591 webserver.cc:460] Webserver started at http://127.0.97.1:36613/ using document root <none> and password file <none>
I20260927 05:05:48.015565 1591 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:48.015652 1591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:48.015916 1591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:48.018950 1591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "7c13c039c7aa4ec388714b1ec0d9a265"
format_stamp: "Formatted at 2026-09-27 05:05:48 on dist-test-slave-hgs0"
server_key: "0fd461984b73a30e66c43c2116dfa0f0"
server_key_iv: "a1ea8d77ee10bdd41aa694b279fbe484"
server_key_version: "encryptionkey@0"
I20260927 05:05:48.019745 1591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7c13c039c7aa4ec388714b1ec0d9a265"
format_stamp: "Formatted at 2026-09-27 05:05:48 on dist-test-slave-hgs0"
server_key: "0fd461984b73a30e66c43c2116dfa0f0"
server_key_iv: "a1ea8d77ee10bdd41aa694b279fbe484"
server_key_version: "encryptionkey@0"
I20260927 05:05:48.024935 1591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:05:48.029064 1606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:48.030442 1591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:05:48.030653 1591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "7c13c039c7aa4ec388714b1ec0d9a265"
format_stamp: "Formatted at 2026-09-27 05:05:48 on dist-test-slave-hgs0"
server_key: "0fd461984b73a30e66c43c2116dfa0f0"
server_key_iv: "a1ea8d77ee10bdd41aa694b279fbe484"
server_key_version: "encryptionkey@0"
I20260927 05:05:48.030907 1591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:48.046983 1591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:48.051359 1591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:48.051663 1591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:48.054404 1591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:48.054534 1591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:48.054693 1591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:48.054775 1591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:05:48 dist-test-slave-hgs0 krb5kdc[1475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:48.084069 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:48.056018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32845 (local address 127.0.97.1:40465)
0927 05:05:48.057030 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:48.057092 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:48.060067 (+ 2975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:48.061112 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:48.061129 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:48.062013 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:48.062890 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:48.062924 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.064757 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.064764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:48.067037 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:48.067057 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.068123 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.068131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:48.068435 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:48.069376 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:48.069416 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:48.073211 (+ 3795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:48.077956 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:48.077976 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:48.078000 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:48.078826 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:48.079384 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:48.079391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:48.079398 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:48.079678 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:48.080581 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:48.080592 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:48.081079 (+ 487us) client_negotiation.cc:769] Sending connection context
0927 05:05:48.081494 (+ 415us) client_negotiation.cc:240] Negotiation successful
0927 05:05:48.082180 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":422,"threads_started":1}
I20260927 05:05:48.085150 1615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:48.055255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40465 (local address 127.0.97.62:32845)
0927 05:05:48.055707 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:48.055713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:48.057237 (+ 1524us) server_negotiation.cc:407] Connection header received
0927 05:05:48.060531 (+ 3294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:48.060540 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:48.060646 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:48.060758 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:48.063167 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.064525 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.067213 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.067831 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.068005 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:48.073517 (+ 5512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:48.073530 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:48.073534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:48.073574 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:48.077660 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:48.079010 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:48.079015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:48.079017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:48.079219 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:48.080043 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:48.080050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:48.080055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:48.080363 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:48.084119 (+ 3756us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:48.084283 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 05:05:48.084896 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260927 05:05:48.107358 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:48.091083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32845 (local address 127.0.97.1:42975)
0927 05:05:48.091760 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:48.091779 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:48.092023 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:48.092374 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:48.092378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:48.093090 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:48.093604 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:48.093623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.095559 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.095563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:48.097025 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:48.097038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.098230 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.098238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:48.098425 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:48.099300 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:48.099320 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:48.100717 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:48.103888 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:48.103893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:48.103898 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:48.104612 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:48.105228 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:48.105235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:48.105242 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:48.105440 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:48.106142 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:48.106151 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:48.106408 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 05:05:48.106615 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 05:05:48.107004 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":375,"threads_started":1}
I20260927 05:05:48.107702 1615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:48.090963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42975 (local address 127.0.97.62:32845)
0927 05:05:48.091132 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:48.091137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:48.091878 (+ 741us) server_negotiation.cc:407] Connection header received
0927 05:05:48.092165 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:48.092175 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:48.092238 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:48.092342 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:48.093819 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.095362 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.097195 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.097966 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.098131 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:48.100941 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:48.100947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:48.100951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:48.101014 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:48.103721 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:48.104824 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:48.104830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:48.104834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:48.104991 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:48.105574 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:48.105578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:48.105580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:48.106003 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:48.106179 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:48.106801 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 05:05:48.107178 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:05:48.111780 1591 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37831
I20260927 05:05:48.111802 1726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37831 every 8 connection(s)
I20260927 05:05:48.113400 1591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:05:48.116053 1727 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32845
I20260927 05:05:48.116330 1727 heartbeater.cc:464] Registering TS with master...
I20260927 05:05:48.117103 1727 heartbeater.cc:511] Master 127.0.97.62:32845 requested a full tablet report, sending...
I20260927 05:05:48.119376 1520 ts_manager.cc:194] Registered new tserver with Master: 7c13c039c7aa4ec388714b1ec0d9a265 (127.0.97.1:37831)
I20260927 05:05:48.120973 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 1591
I20260927 05:05:48.121187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:05:48.121337 1520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40465
I20260927 05:05:48.121981 388 external_mini_cluster.cc:1428] Setting key 25fe4bb2615989244cee160b3cf58ada
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:05:48.161619 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32845
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:48.518865 1731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:48.519344 1731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:48.519513 1731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:48.527323 1731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:48.527446 1731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:48.527534 1731 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:05:48.527616 1731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:05:48.538017 1731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:32845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:48.539299 1731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:48.540710 1731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:48.556245 1737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:48.556288 1736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:48.556313 1739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:48.557117 1731 server_base.cc:1034] running on GCE node
I20260927 05:05:48.557788 1731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:48.558815 1731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:48.560060 1731 hybrid_clock.cc:648] HybridClock initialized: now 1790485548559968 us; error 67 us; skew 500 ppm
I20260927 05:05:48.560381 1731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:48 dist-test-slave-hgs0 krb5kdc[1475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:48.563656 1731 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:05:48.565330 1731 webserver.cc:460] Webserver started at http://127.0.97.2:46393/ using document root <none> and password file <none>
I20260927 05:05:48.566110 1731 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:48.566237 1731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:48.566588 1731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:48.569675 1731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "a18149013f314502afe1474c26f2cee2"
format_stamp: "Formatted at 2026-09-27 05:05:48 on dist-test-slave-hgs0"
server_key: "876ef5c96cdfa8a09902e8ff23aa1636"
server_key_iv: "dda8124c2bb9b7931f3c7fa9c57d49b2"
server_key_version: "encryptionkey@0"
I20260927 05:05:48.570530 1731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a18149013f314502afe1474c26f2cee2"
format_stamp: "Formatted at 2026-09-27 05:05:48 on dist-test-slave-hgs0"
server_key: "876ef5c96cdfa8a09902e8ff23aa1636"
server_key_iv: "dda8124c2bb9b7931f3c7fa9c57d49b2"
server_key_version: "encryptionkey@0"
I20260927 05:05:48.576013 1731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:05:48.579737 1746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:48.581254 1731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:05:48.581450 1731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "a18149013f314502afe1474c26f2cee2"
format_stamp: "Formatted at 2026-09-27 05:05:48 on dist-test-slave-hgs0"
server_key: "876ef5c96cdfa8a09902e8ff23aa1636"
server_key_iv: "dda8124c2bb9b7931f3c7fa9c57d49b2"
server_key_version: "encryptionkey@0"
I20260927 05:05:48.581710 1731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:48.617481 1731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:48.621833 1731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:48.622133 1731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:48.625032 1731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:48.625147 1731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:48.625324 1731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:48.625402 1731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:05:48 dist-test-slave-hgs0 krb5kdc[1475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:48.654037 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:48.625664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37701 (local address 127.0.97.62:32845)
0927 05:05:48.626298 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:48.626304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:48.627635 (+ 1331us) server_negotiation.cc:407] Connection header received
0927 05:05:48.630160 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:48.630168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:48.630261 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:48.630408 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:48.633301 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.634618 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.638385 (+ 3767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.639261 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.639441 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:48.645605 (+ 6164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:48.645614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:48.645622 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:48.645677 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:48.650049 (+ 4372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:48.651278 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:48.651288 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:48.651294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:48.651472 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:48.652117 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:48.652125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:48.652128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:48.652472 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:48.652631 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:48.653412 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 05:05:48.653640 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":397,"threads_started":1}
I20260927 05:05:48.655206 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:48.626424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32845 (local address 127.0.97.2:37701)
0927 05:05:48.627418 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:48.627473 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:48.629814 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:48.630754 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:48.630774 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:48.631813 (+ 1039us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:48.633068 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:48.633108 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.634950 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.634963 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:48.638177 (+ 3214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:48.638192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.639871 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.639883 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:48.640404 (+ 521us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:48.641510 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:48.641559 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:48.645251 (+ 3692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:48.650309 (+ 5058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:48.650318 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:48.650334 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:48.651105 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:48.651630 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:48.651636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:48.651646 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:48.651911 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:48.652621 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:48.652630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:48.653023 (+ 393us) client_negotiation.cc:769] Sending connection context
0927 05:05:48.653303 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 05:05:48.653879 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":427,"threads_started":1}
I20260927 05:05:48.679806 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:48.662063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41161 (local address 127.0.97.62:32845)
0927 05:05:48.662286 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:48.662295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:48.663112 (+ 817us) server_negotiation.cc:407] Connection header received
0927 05:05:48.663340 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:48.663345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:48.663419 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:48.663527 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:48.665485 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.666304 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.669337 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.669961 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.670129 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:48.672810 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:48.672817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:48.672828 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:48.672863 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:48.676503 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:48.677568 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:48.677576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:48.677582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:48.677753 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:48.678203 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:48.678208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:48.678212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:48.678610 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:48.678773 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:48.679208 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 05:05:48.679440 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:05:48.679924 1828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:48.662079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32845 (local address 127.0.97.2:41161)
0927 05:05:48.662933 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:48.662963 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:48.663193 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:48.663602 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:48.663610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:48.664614 (+ 1004us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:48.665248 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:48.665273 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.666491 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.666500 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:48.668994 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:48.669031 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:48.670456 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:48.670466 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:48.670702 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:48.671511 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:48.671547 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:48.672600 (+ 1053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:48.676683 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:48.676689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:48.676693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:48.677409 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:48.677911 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:48.677917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:48.677921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:48.678069 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:48.678738 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:48.678744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:48.678926 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 05:05:48.679104 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 05:05:48.679448 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":423,"threads_started":1}
I20260927 05:05:48.683171 1731 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37359
I20260927 05:05:48.683218 1866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37359 every 8 connection(s)
I20260927 05:05:48.684499 1731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:05:48.686545 1867 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32845
I20260927 05:05:48.686833 1867 heartbeater.cc:464] Registering TS with master...
I20260927 05:05:48.687530 1867 heartbeater.cc:511] Master 127.0.97.62:32845 requested a full tablet report, sending...
I20260927 05:05:48.689028 1520 ts_manager.cc:194] Registered new tserver with Master: a18149013f314502afe1474c26f2cee2 (127.0.97.2:37359)
I20260927 05:05:48.689502 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 1731
I20260927 05:05:48.689703 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:05:48.689949 1520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37701
I20260927 05:05:48.690183 388 external_mini_cluster.cc:1428] Setting key ad44dfe346f5828ab328c2d509803c1c
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:05:48.733737 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32845
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:49.080919 1871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:49.081224 1871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:49.081394 1871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:49.089002 1871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:49.089125 1871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:49.089310 1871 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:05:49.089442 1871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:05:49.099423 1871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:32845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:49.101037 1871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:49.102515 1871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:49.117157 1877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:49.117264 1876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:49.117416 1879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:49.118206 1871 server_base.cc:1034] running on GCE node
I20260927 05:05:49.118882 1871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:49.119856 1871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:49.121099 1871 hybrid_clock.cc:648] HybridClock initialized: now 1790485549121004 us; error 63 us; skew 500 ppm
I20260927 05:05:49.121503 1871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:49 dist-test-slave-hgs0 krb5kdc[1475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:49.124882 1727 heartbeater.cc:503] Master 127.0.97.62:32845 was elected leader, sending a full tablet report...
I20260927 05:05:49.125128 1871 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:05:49.127064 1871 webserver.cc:460] Webserver started at http://127.0.97.3:45853/ using document root <none> and password file <none>
I20260927 05:05:49.127868 1871 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:49.128026 1871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:49.128381 1871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:49.131527 1871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "f3e9559d45ee4e06a8bbcc04d227f47a"
format_stamp: "Formatted at 2026-09-27 05:05:49 on dist-test-slave-hgs0"
server_key: "61fd346e8fabe91b0c95f5862fe9e9b4"
server_key_iv: "41c8e665eff6052cf6f960f9e33a5e3c"
server_key_version: "encryptionkey@0"
I20260927 05:05:49.132401 1871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f3e9559d45ee4e06a8bbcc04d227f47a"
format_stamp: "Formatted at 2026-09-27 05:05:49 on dist-test-slave-hgs0"
server_key: "61fd346e8fabe91b0c95f5862fe9e9b4"
server_key_iv: "41c8e665eff6052cf6f960f9e33a5e3c"
server_key_version: "encryptionkey@0"
I20260927 05:05:49.138080 1871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:05:49.141695 1886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:49.143069 1871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:05:49.143254 1871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "f3e9559d45ee4e06a8bbcc04d227f47a"
format_stamp: "Formatted at 2026-09-27 05:05:49 on dist-test-slave-hgs0"
server_key: "61fd346e8fabe91b0c95f5862fe9e9b4"
server_key_iv: "41c8e665eff6052cf6f960f9e33a5e3c"
server_key_version: "encryptionkey@0"
I20260927 05:05:49.143482 1871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:49.158108 1871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:49.162613 1871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:49.162887 1871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:49.165745 1871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:49.165846 1871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:49.166046 1871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:49.166119 1871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 05:05:49 dist-test-slave-hgs0 krb5kdc[1475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:49.194379 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.166192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55851 (local address 127.0.97.62:32845)
0927 05:05:49.166376 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.166383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.167884 (+ 1501us) server_negotiation.cc:407] Connection header received
0927 05:05:49.170060 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.170066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.170162 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.170275 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:49.172644 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.173484 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.176925 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.177517 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.177693 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.183832 (+ 6139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:49.183838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:49.183842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:49.183882 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:49.187796 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:49.189664 (+ 1868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:49.189678 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:49.189687 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:49.189938 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:49.191068 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:49.191077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:49.191083 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:49.191561 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:49.191785 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.193659 (+ 1874us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.194040 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:05:49.196470 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.166883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32845 (local address 127.0.97.3:55851)
0927 05:05:49.167728 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:49.167765 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:49.169704 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:49.170596 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:49.170611 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:49.171457 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:49.172442 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.172476 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.173692 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.173699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.176750 (+ 3051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.176767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.177811 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.177825 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.178257 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.179288 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:49.179330 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:49.183455 (+ 4125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:49.188087 (+ 4632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:49.188107 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:49.188145 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:49.189344 (+ 1199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:49.190213 (+ 869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:49.190223 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:49.190227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:49.190748 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:49.191769 (+ 1021us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:49.191780 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:49.192460 (+ 680us) client_negotiation.cc:769] Sending connection context
0927 05:05:49.193052 (+ 592us) client_negotiation.cc:240] Negotiation successful
0927 05:05:49.193998 (+ 946us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":349,"threads_started":1}
I20260927 05:05:49.222395 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.203081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39113 (local address 127.0.97.62:32845)
0927 05:05:49.203313 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.203322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.204040 (+ 718us) server_negotiation.cc:407] Connection header received
0927 05:05:49.204354 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.204363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.204456 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.204589 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:49.206366 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.207304 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.209759 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.210459 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.210644 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.213388 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:49.213407 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:49.213411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:49.213450 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:49.218183 (+ 4733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:49.219430 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:49.219435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:49.219438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:49.219649 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:49.220282 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:49.220287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:49.220290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:49.220682 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:49.220845 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.221517 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.221844 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 05:05:49.223302 1958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.203052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32845 (local address 127.0.97.3:39113)
0927 05:05:49.203875 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:49.203903 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:49.204129 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:49.204802 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:49.204809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:49.205608 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:49.206187 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.206200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.207552 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.207562 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.209564 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.209582 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.210688 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.210695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.210868 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.211767 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:49.211803 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:49.213129 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:49.218421 (+ 5292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:49.218427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:49.218437 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:49.219207 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:49.219814 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:49.219823 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:49.219827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:49.220107 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:49.220823 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:49.220827 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:49.221134 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 05:05:49.221323 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 05:05:49.222953 (+ 1630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":423,"threads_started":1}
I20260927 05:05:49.228296 1871 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46355
I20260927 05:05:49.228334 2005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46355 every 8 connection(s)
I20260927 05:05:49.229604 1871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:05:49.231858 2006 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32845
I20260927 05:05:49.232172 2006 heartbeater.cc:464] Registering TS with master...
I20260927 05:05:49.232887 2006 heartbeater.cc:511] Master 127.0.97.62:32845 requested a full tablet report, sending...
I20260927 05:05:49.234635 1520 ts_manager.cc:194] Registered new tserver with Master: f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355)
I20260927 05:05:49.235105 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 1871
I20260927 05:05:49.235297 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:05:49.235924 388 external_mini_cluster.cc:1428] Setting key 4bd71e44a581c33126bfdfac05c3c39e
I20260927 05:05:49.235877 1520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55851
I20260927 05:05:49.239434 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 05:05:49 dist-test-slave-hgs0 krb5kdc[1475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 05:05:49 dist-test-slave-hgs0 krb5kdc[1475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:49.289453 1755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.274324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.0.97.62:32845)
0927 05:05:49.274582 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.274590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.274761 (+ 171us) server_negotiation.cc:407] Connection header received
0927 05:05:49.274922 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.274926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.275013 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.275108 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:49.276681 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.277457 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.278839 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.279285 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.279398 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.283748 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:49.283754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:49.283758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:49.283800 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:49.286950 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:49.287629 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:49.287633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:49.287636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:49.287770 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:49.288235 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:49.288239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:49.288241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:49.288571 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:49.288759 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.289008 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.289188 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 05:05:49.293920 1520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:05:49.301234 1520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:05:49.331832 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.314699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46355 (local address 127.0.0.1:37592)
0927 05:05:49.316019 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:49.316069 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:49.316286 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:49.319646 (+ 3360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:49.319659 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:49.319706 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:49.320259 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.320276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.325801 (+ 5525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.325810 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.328732 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.328748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.330734 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.330741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.330988 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.331058 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 05:05:49.331195 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 05:05:49.331450 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"thread_start_us":13820,"threads_started":1}
I20260927 05:05:49.331892 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.315416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.0.97.3:46355)
0927 05:05:49.317144 (+ 1728us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.317158 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.317202 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:05:49.318846 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.318860 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.319284 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.320550 (+ 1266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:49.320957 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.325056 (+ 4099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.328936 (+ 3880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.330521 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.330688 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.330969 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.331398 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.331583 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1464,"thread_start_us":1249,"threads_started":1}
I20260927 05:05:49.336166 1936 tablet_service.cc:1467] Processing CreateTablet for tablet b17f87fb3c9e446d94baf493d5608fdd (DEFAULT_TABLE table=test-table [id=3187b878d3bb4216817648ec283b1628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:49.337075 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.315278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.0.97.2:37359)
0927 05:05:49.325742 (+ 10464us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.325754 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.328313 (+ 2559us) server_negotiation.cc:407] Connection header received
0927 05:05:49.328539 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.328548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.328904 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.329165 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:49.329836 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.331346 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.334946 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.335988 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.336143 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.336360 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.336606 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.336759 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10279,"thread_start_us":10135,"threads_started":1}
I20260927 05:05:49.337390 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.314853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37359 (local address 127.0.0.1:46510)
0927 05:05:49.328132 (+ 13279us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:49.328153 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:49.328302 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:49.329143 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:49.329148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:49.329170 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:49.329618 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.329631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.332679 (+ 3048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.332684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.334793 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.334812 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.336194 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.336200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.336364 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.336402 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 05:05:49.336551 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 05:05:49.336689 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13138,"thread_start_us":599,"threads_started":1}
I20260927 05:05:49.337419 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.316428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37831 (local address 127.0.0.1:44956)
0927 05:05:49.328864 (+ 12436us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:49.328882 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:49.329003 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:49.329683 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:49.329688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:49.329717 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:49.330079 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.330090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.331933 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.331938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.334243 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.334259 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.336572 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.336578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.336747 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.336777 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 05:05:49.336884 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 05:05:49.337025 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12285,"mutex_wait_us":2,"thread_start_us":11406,"threads_started":1}
I20260927 05:05:49.337821 1936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17f87fb3c9e446d94baf493d5608fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:49.337973 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.317020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.0.97.1:37831)
0927 05:05:49.327568 (+ 10548us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.327578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.329006 (+ 1428us) server_negotiation.cc:407] Connection header received
0927 05:05:49.329138 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.329146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.329429 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.329673 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:49.330268 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.331768 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.334437 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.336064 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.336207 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.336410 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.337518 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.337686 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10353,"thread_start_us":10202,"threads_started":1}
I20260927 05:05:49.341384 1797 tablet_service.cc:1467] Processing CreateTablet for tablet b17f87fb3c9e446d94baf493d5608fdd (DEFAULT_TABLE table=test-table [id=3187b878d3bb4216817648ec283b1628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:49.341836 1657 tablet_service.cc:1467] Processing CreateTablet for tablet b17f87fb3c9e446d94baf493d5608fdd (DEFAULT_TABLE table=test-table [id=3187b878d3bb4216817648ec283b1628]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:49.342624 1797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17f87fb3c9e446d94baf493d5608fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:49.343147 1657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17f87fb3c9e446d94baf493d5608fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:49.353286 2025 tablet_bootstrap.cc:492] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: Bootstrap starting.
I20260927 05:05:49.357538 2027 tablet_bootstrap.cc:492] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: Bootstrap starting.
I20260927 05:05:49.359411 2026 tablet_bootstrap.cc:492] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: Bootstrap starting.
I20260927 05:05:49.359463 2025 tablet_bootstrap.cc:654] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:49.360868 2025 log.cc:824] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:49.361457 2027 tablet_bootstrap.cc:654] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:49.362838 2027 log.cc:824] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:49.362941 2026 tablet_bootstrap.cc:654] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:49.364178 2026 log.cc:824] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:49.364280 2025 tablet_bootstrap.cc:492] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: No bootstrap required, opened a new log
I20260927 05:05:49.364578 2025 ts_tablet_manager.cc:1397] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 05:05:49.366493 2027 tablet_bootstrap.cc:492] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: No bootstrap required, opened a new log
I20260927 05:05:49.366878 2027 ts_tablet_manager.cc:1397] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 05:05:49.373711 2026 tablet_bootstrap.cc:492] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: No bootstrap required, opened a new log
I20260927 05:05:49.374043 2026 ts_tablet_manager.cc:1397] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20260927 05:05:49.373878 2025 raft_consensus.cc:348] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } }
I20260927 05:05:49.374372 2025 raft_consensus.cc:374] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:49.374500 2025 raft_consensus.cc:729] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e9559d45ee4e06a8bbcc04d227f47a, State: Initialized, Role: FOLLOWER
I20260927 05:05:49.375173 2025 consensus_queue.cc:260] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } }
I20260927 05:05:49.376464 2006 heartbeater.cc:503] Master 127.0.97.62:32845 was elected leader, sending a full tablet report...
I20260927 05:05:49.377040 2025 ts_tablet_manager.cc:1428] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260927 05:05:49.378718 2027 raft_consensus.cc:348] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } }
I20260927 05:05:49.379136 2027 raft_consensus.cc:374] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:49.379238 2027 raft_consensus.cc:729] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c13c039c7aa4ec388714b1ec0d9a265, State: Initialized, Role: FOLLOWER
I20260927 05:05:49.379897 2027 consensus_queue.cc:260] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } }
I20260927 05:05:49.383854 2027 ts_tablet_manager.cc:1428] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
I20260927 05:05:49.385414 2026 raft_consensus.cc:348] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } }
I20260927 05:05:49.385947 2026 raft_consensus.cc:374] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:49.386085 2026 raft_consensus.cc:729] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18149013f314502afe1474c26f2cee2, State: Initialized, Role: FOLLOWER
I20260927 05:05:49.386772 2026 consensus_queue.cc:260] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } }
I20260927 05:05:49.388039 1867 heartbeater.cc:503] Master 127.0.97.62:32845 was elected leader, sending a full tablet report...
I20260927 05:05:49.388652 2026 ts_tablet_manager.cc:1428] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260927 05:05:49.436635 1868 tablet.cc:2369] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:05:49.481372 2007 tablet.cc:2369] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:05:49.616323 1728 tablet.cc:2369] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:05:49.619755 2032 raft_consensus.cc:482] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:05:49.620078 2032 raft_consensus.cc:504] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } }
I20260927 05:05:49.621824 2032 leader_election.cc:288] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355), a18149013f314502afe1474c26f2cee2 (127.0.97.2:37359)
I20260927 05:05:49.629192 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.622368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33345 (local address 127.0.97.3:46355)
0927 05:05:49.622623 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.622632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.623407 (+ 775us) server_negotiation.cc:407] Connection header received
0927 05:05:49.623601 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.623606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.623663 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.623756 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:49.624877 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.625584 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.627372 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.628203 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.628359 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.628482 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.628771 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.628903 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:05:49.629202 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.622486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53823 (local address 127.0.97.2:37359)
0927 05:05:49.622730 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.622739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.623345 (+ 606us) server_negotiation.cc:407] Connection header received
0927 05:05:49.623598 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.623602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.623659 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.623759 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:49.624764 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.625485 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.627382 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.628219 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.628371 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.628491 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.628785 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.628919 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:05:49.629367 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.622311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37359 (local address 127.0.97.1:53823)
0927 05:05:49.623220 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:49.623235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:49.623421 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:49.623772 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:49.623777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:49.623809 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:49.624602 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.624613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.625616 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.625625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.627224 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.627235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.628495 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.628504 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.628633 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.628655 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:05:49.628758 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 05:05:49.628919 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":74,"thread_start_us":516,"threads_started":1}
I20260927 05:05:49.629843 2035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.622338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46355 (local address 127.0.97.1:33345)
0927 05:05:49.623287 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:49.623301 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:49.623451 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:49.623781 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:49.623785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:49.623872 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:49.624709 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.624718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.625729 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.625735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.627202 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.627216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.628440 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.628447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.628607 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.628634 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 05:05:49.628744 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 05:05:49.628903 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":27,"spinlock_wait_cycles":26880,"thread_start_us":630,"threads_started":1}
I20260927 05:05:49.630507 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17f87fb3c9e446d94baf493d5608fdd" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18149013f314502afe1474c26f2cee2" is_pre_election: true
I20260927 05:05:49.630854 1961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17f87fb3c9e446d94baf493d5608fdd" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" is_pre_election: true
I20260927 05:05:49.631026 1817 raft_consensus.cc:2454] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 0.
I20260927 05:05:49.631385 1961 raft_consensus.cc:2454] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 0.
I20260927 05:05:49.631762 1608 leader_election.cc:302] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c13c039c7aa4ec388714b1ec0d9a265, a18149013f314502afe1474c26f2cee2; no voters:
I20260927 05:05:49.632272 2032 raft_consensus.cc:2781] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:05:49.632416 2032 raft_consensus.cc:482] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:05:49.632541 2032 raft_consensus.cc:3037] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:49.635461 2032 raft_consensus.cc:504] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } }
I20260927 05:05:49.636233 2032 leader_election.cc:288] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 election: Requested vote from peers f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355), a18149013f314502afe1474c26f2cee2 (127.0.97.2:37359)
I20260927 05:05:49.636616 1961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17f87fb3c9e446d94baf493d5608fdd" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a"
I20260927 05:05:49.636829 1961 raft_consensus.cc:3037] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:49.636765 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17f87fb3c9e446d94baf493d5608fdd" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18149013f314502afe1474c26f2cee2"
I20260927 05:05:49.637030 1817 raft_consensus.cc:3037] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:49.639230 1961 raft_consensus.cc:2454] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 1.
I20260927 05:05:49.639417 1817 raft_consensus.cc:2454] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 1.
I20260927 05:05:49.639763 1610 leader_election.cc:302] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c13c039c7aa4ec388714b1ec0d9a265, f3e9559d45ee4e06a8bbcc04d227f47a; no voters:
I20260927 05:05:49.640115 2032 raft_consensus.cc:2781] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:49.640571 2032 raft_consensus.cc:686] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 LEADER]: Becoming Leader. State: Replica: 7c13c039c7aa4ec388714b1ec0d9a265, State: Running, Role: LEADER
I20260927 05:05:49.641158 2032 consensus_queue.cc:237] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } }
I20260927 05:05:49.645797 1520 catalog_manager.cc:5495] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c13c039c7aa4ec388714b1ec0d9a265 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } health_report { overall_health: HEALTHY } } }
I20260927 05:05:49.704401 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.698152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.0.97.1:37831)
0927 05:05:49.698386 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.698391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.698577 (+ 186us) server_negotiation.cc:407] Connection header received
0927 05:05:49.698923 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.698928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.699044 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.699190 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:49.699572 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.700413 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.701940 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.702374 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.702477 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.703076 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:49.703714 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:49.703843 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.704053 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.704130 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:05:49.713550 1961 raft_consensus.cc:1261] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a [term 1 FOLLOWER]: Refusing update from remote peer 7c13c039c7aa4ec388714b1ec0d9a265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:49.713550 1817 raft_consensus.cc:1261] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Refusing update from remote peer 7c13c039c7aa4ec388714b1ec0d9a265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:49.714587 2036 consensus_queue.cc:1035] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:49.714972 2032 consensus_queue.cc:1035] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:49.748914 1519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:49.749249 1519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:49.753219 1519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a: Sending DeleteTablet for 3 replicas of tablet b17f87fb3c9e446d94baf493d5608fdd
I20260927 05:05:49.754556 1936 tablet_service.cc:1514] Processing DeleteTablet for tablet b17f87fb3c9e446d94baf493d5608fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260927 05:05:49.754735 1797 tablet_service.cc:1514] Processing DeleteTablet for tablet b17f87fb3c9e446d94baf493d5608fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46510
I20260927 05:05:49.755003 1657 tablet_service.cc:1514] Processing DeleteTablet for tablet b17f87fb3c9e446d94baf493d5608fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44956
I20260927 05:05:49.756248 2049 tablet_replica.cc:330] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: stopping tablet replica
I20260927 05:05:49.757022 2049 raft_consensus.cc:2229] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:49.757737 2050 tablet_replica.cc:330] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: stopping tablet replica
I20260927 05:05:49.757928 2049 raft_consensus.cc:2258] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:49.758196 2050 raft_consensus.cc:2229] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:05:49.758093 1519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:05:49.758893 1519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:05:49.758862 2050 raft_consensus.cc:2258] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:49.761268 2049 ts_tablet_manager.cc:1905] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:49.762907 2050 ts_tablet_manager.cc:1905] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:49.768682 1796 tablet_service.cc:1467] Processing CreateTablet for tablet 62342c9235d84ab0bc50bf688a9f71c2 (DEFAULT_TABLE table=test-table [id=6099a48649094d95b8b5dcf3347eb14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:49.769438 2049 ts_tablet_manager.cc:1918] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:05:49.769734 2049 log.cc:1196] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/b17f87fb3c9e446d94baf493d5608fdd
I20260927 05:05:49.769769 1796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62342c9235d84ab0bc50bf688a9f71c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:49.770375 2049 ts_tablet_manager.cc:1939] T b17f87fb3c9e446d94baf493d5608fdd P f3e9559d45ee4e06a8bbcc04d227f47a: Deleting consensus metadata
I20260927 05:05:49.772351 1936 tablet_service.cc:1467] Processing CreateTablet for tablet 62342c9235d84ab0bc50bf688a9f71c2 (DEFAULT_TABLE table=test-table [id=6099a48649094d95b8b5dcf3347eb14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:49.772821 1936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62342c9235d84ab0bc50bf688a9f71c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:49.776363 1507 catalog_manager.cc:4841] TS f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355): tablet b17f87fb3c9e446d94baf493d5608fdd (table test-table [id=3187b878d3bb4216817648ec283b1628]) successfully deleted
I20260927 05:05:49.776444 2051 tablet_replica.cc:330] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: stopping tablet replica
I20260927 05:05:49.776641 1657 tablet_service.cc:1467] Processing CreateTablet for tablet 62342c9235d84ab0bc50bf688a9f71c2 (DEFAULT_TABLE table=test-table [id=6099a48649094d95b8b5dcf3347eb14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:49.776929 2051 raft_consensus.cc:2229] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:49.777129 1657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62342c9235d84ab0bc50bf688a9f71c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:49.777472 2051 raft_consensus.cc:2258] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:49.778467 2050 ts_tablet_manager.cc:1918] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:05:49.778658 2050 log.cc:1196] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/b17f87fb3c9e446d94baf493d5608fdd
I20260927 05:05:49.779260 2050 ts_tablet_manager.cc:1939] T b17f87fb3c9e446d94baf493d5608fdd P 7c13c039c7aa4ec388714b1ec0d9a265: Deleting consensus metadata
I20260927 05:05:49.781227 2025 tablet_bootstrap.cc:492] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a: Bootstrap starting.
I20260927 05:05:49.781190 1505 catalog_manager.cc:4841] TS 7c13c039c7aa4ec388714b1ec0d9a265 (127.0.97.1:37831): tablet b17f87fb3c9e446d94baf493d5608fdd (table test-table [id=3187b878d3bb4216817648ec283b1628]) successfully deleted
I20260927 05:05:49.783336 2025 tablet_bootstrap.cc:654] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:49.783624 2051 ts_tablet_manager.cc:1905] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:49.789790 2027 tablet_bootstrap.cc:492] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265: Bootstrap starting.
I20260927 05:05:49.790462 2026 tablet_bootstrap.cc:492] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2: Bootstrap starting.
I20260927 05:05:49.791955 2027 tablet_bootstrap.cc:654] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:49.792533 2026 tablet_bootstrap.cc:654] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:49.797435 2051 ts_tablet_manager.cc:1918] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:05:49.797612 2051 log.cc:1196] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/b17f87fb3c9e446d94baf493d5608fdd
I20260927 05:05:49.798182 2051 ts_tablet_manager.cc:1939] T b17f87fb3c9e446d94baf493d5608fdd P a18149013f314502afe1474c26f2cee2: Deleting consensus metadata
I20260927 05:05:49.799978 2027 tablet_bootstrap.cc:492] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265: No bootstrap required, opened a new log
I20260927 05:05:49.800034 2026 tablet_bootstrap.cc:492] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2: No bootstrap required, opened a new log
I20260927 05:05:49.800154 2027 ts_tablet_manager.cc:1397] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260927 05:05:49.800216 1505 catalog_manager.cc:4841] TS a18149013f314502afe1474c26f2cee2 (127.0.97.2:37359): tablet b17f87fb3c9e446d94baf493d5608fdd (table test-table [id=3187b878d3bb4216817648ec283b1628]) successfully deleted
I20260927 05:05:49.800323 2026 ts_tablet_manager.cc:1397] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260927 05:05:49.801237 2025 tablet_bootstrap.cc:492] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a: No bootstrap required, opened a new log
I20260927 05:05:49.801119 2027 raft_consensus.cc:348] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.801369 2027 raft_consensus.cc:374] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:49.801467 2027 raft_consensus.cc:729] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c13c039c7aa4ec388714b1ec0d9a265, State: Initialized, Role: FOLLOWER
I20260927 05:05:49.801632 2025 ts_tablet_manager.cc:1397] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.013s
I20260927 05:05:49.801821 2026 raft_consensus.cc:348] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.802052 2026 raft_consensus.cc:374] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:49.802141 2026 raft_consensus.cc:729] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18149013f314502afe1474c26f2cee2, State: Initialized, Role: FOLLOWER
I20260927 05:05:49.801767 2027 consensus_queue.cc:260] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.802403 2026 consensus_queue.cc:260] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.803059 2025 raft_consensus.cc:348] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.803339 2025 raft_consensus.cc:374] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:49.803483 2025 raft_consensus.cc:729] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e9559d45ee4e06a8bbcc04d227f47a, State: Initialized, Role: FOLLOWER
I20260927 05:05:49.803791 2025 consensus_queue.cc:260] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.805225 2025 ts_tablet_manager.cc:1428] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 05:05:49.806037 2027 ts_tablet_manager.cc:1428] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260927 05:05:49.806133 2026 ts_tablet_manager.cc:1428] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260927 05:05:49.818044 2044 raft_consensus.cc:482] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:05:49.818269 2044 raft_consensus.cc:504] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.819499 2044 leader_election.cc:288] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a18149013f314502afe1474c26f2cee2 (127.0.97.2:37359), f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355)
I20260927 05:05:49.820116 1961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62342c9235d84ab0bc50bf688a9f71c2" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" is_pre_election: true
I20260927 05:05:49.820369 1961 raft_consensus.cc:2454] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 0.
I20260927 05:05:49.820624 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62342c9235d84ab0bc50bf688a9f71c2" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18149013f314502afe1474c26f2cee2" is_pre_election: true
I20260927 05:05:49.821082 1610 leader_election.cc:302] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c13c039c7aa4ec388714b1ec0d9a265, f3e9559d45ee4e06a8bbcc04d227f47a; no voters:
I20260927 05:05:49.821231 1817 raft_consensus.cc:2454] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 0.
I20260927 05:05:49.821497 2044 raft_consensus.cc:2781] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:05:49.821633 2044 raft_consensus.cc:482] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:05:49.821727 2044 raft_consensus.cc:3037] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:49.824263 2044 raft_consensus.cc:504] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.824863 2044 leader_election.cc:288] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 election: Requested vote from peers a18149013f314502afe1474c26f2cee2 (127.0.97.2:37359), f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355)
I20260927 05:05:49.825452 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62342c9235d84ab0bc50bf688a9f71c2" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18149013f314502afe1474c26f2cee2"
I20260927 05:05:49.825489 1961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62342c9235d84ab0bc50bf688a9f71c2" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a"
I20260927 05:05:49.825672 1817 raft_consensus.cc:3037] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:49.825742 1961 raft_consensus.cc:3037] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:49.828161 1817 raft_consensus.cc:2454] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 1.
I20260927 05:05:49.828604 1608 leader_election.cc:302] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c13c039c7aa4ec388714b1ec0d9a265, a18149013f314502afe1474c26f2cee2; no voters:
I20260927 05:05:49.828719 1961 raft_consensus.cc:2454] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 1.
I20260927 05:05:49.828958 2044 raft_consensus.cc:2781] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:49.829183 2044 raft_consensus.cc:686] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 LEADER]: Becoming Leader. State: Replica: 7c13c039c7aa4ec388714b1ec0d9a265, State: Running, Role: LEADER
I20260927 05:05:49.829524 2044 consensus_queue.cc:237] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.832536 1519 catalog_manager.cc:5495] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c13c039c7aa4ec388714b1ec0d9a265 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } health_report { overall_health: UNKNOWN } } }
Sep 27 05:05:49 dist-test-slave-hgs0 krb5kdc[1475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:49.892367 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.878488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.0.97.62:32845)
0927 05:05:49.879408 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.879413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.879435 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:05:49.879487 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.879490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.879546 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.879657 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:49.880783 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.881557 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.882810 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.883230 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.883372 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.887060 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:49.887065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:49.887068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:49.887115 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:49.889690 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:49.890360 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:49.890368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:49.890374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:49.890491 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:49.890835 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:49.890842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:49.890846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:49.891103 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:49.891247 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.891664 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.891918 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":799,"thread_start_us":649,"threads_started":1}
I20260927 05:05:49.892417 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.878393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32845 (local address 127.0.0.1:42054)
0927 05:05:49.879098 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:49.879139 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:49.879322 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:49.879683 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:49.879687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:49.880431 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:49.880638 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.880647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.881689 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.881696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.882672 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.882683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.883394 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.883400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.883583 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.884331 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:49.884348 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:49.886868 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:49.889819 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:49.889834 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:49.889842 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:49.890246 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:49.890592 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:49.890599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:49.890603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:49.890735 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:49.891224 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:49.891233 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:49.891405 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 05:05:49.891585 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 05:05:49.892047 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":157,"thread_start_us":378,"threads_started":1}
I20260927 05:05:49.897830 1519 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42054:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 05:05:49.899308 1519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:05:49.906345 1657 tablet_service.cc:1467] Processing CreateTablet for tablet de33f560b83f477f92ca45efd0442cca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5adae707a6e4436bb92caf9a3313a707]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:05:49.906364 1796 tablet_service.cc:1467] Processing CreateTablet for tablet de33f560b83f477f92ca45efd0442cca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5adae707a6e4436bb92caf9a3313a707]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:05:49.906836 1657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de33f560b83f477f92ca45efd0442cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:49.906912 1796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de33f560b83f477f92ca45efd0442cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:49.908478 1936 tablet_service.cc:1467] Processing CreateTablet for tablet de33f560b83f477f92ca45efd0442cca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5adae707a6e4436bb92caf9a3313a707]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:05:49.908990 1936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de33f560b83f477f92ca45efd0442cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:49.913995 2027 tablet_bootstrap.cc:492] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: Bootstrap starting.
I20260927 05:05:49.914783 2026 tablet_bootstrap.cc:492] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: Bootstrap starting.
I20260927 05:05:49.915740 2027 tablet_bootstrap.cc:654] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:49.916131 2025 tablet_bootstrap.cc:492] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: Bootstrap starting.
I20260927 05:05:49.916332 2026 tablet_bootstrap.cc:654] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:49.917691 2025 tablet_bootstrap.cc:654] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:49.928777 2026 tablet_bootstrap.cc:492] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: No bootstrap required, opened a new log
I20260927 05:05:49.928982 2026 ts_tablet_manager.cc:1397] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260927 05:05:49.929644 2026 raft_consensus.cc:348] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.929877 2026 raft_consensus.cc:374] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:49.929975 2026 raft_consensus.cc:729] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18149013f314502afe1474c26f2cee2, State: Initialized, Role: FOLLOWER
I20260927 05:05:49.930270 2026 consensus_queue.cc:260] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.930735 2033 tablet_replica.cc:439] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } }
I20260927 05:05:49.930967 2026 ts_tablet_manager.cc:1428] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 05:05:49.931142 2033 tablet_replica.cc:442] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:49.937500 2025 tablet_bootstrap.cc:492] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: No bootstrap required, opened a new log
I20260927 05:05:49.937732 2025 ts_tablet_manager.cc:1397] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.000s
I20260927 05:05:49.938335 2027 tablet_bootstrap.cc:492] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: No bootstrap required, opened a new log
I20260927 05:05:49.938565 2027 ts_tablet_manager.cc:1397] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.009s
I20260927 05:05:49.938589 2025 raft_consensus.cc:348] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.938875 2025 raft_consensus.cc:374] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:49.938962 2025 raft_consensus.cc:729] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e9559d45ee4e06a8bbcc04d227f47a, State: Initialized, Role: FOLLOWER
I20260927 05:05:49.939297 2025 consensus_queue.cc:260] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.939415 2027 raft_consensus.cc:348] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.939697 2027 raft_consensus.cc:374] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:49.939815 2027 raft_consensus.cc:729] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c13c039c7aa4ec388714b1ec0d9a265, State: Initialized, Role: FOLLOWER
I20260927 05:05:49.939829 2031 tablet_replica.cc:439] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } }
I20260927 05:05:49.940217 2031 tablet_replica.cc:442] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:49.940116 2027 consensus_queue.cc:260] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.940634 2044 tablet_replica.cc:439] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } }
I20260927 05:05:49.940825 2027 ts_tablet_manager.cc:1428] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:05:49.941606 2044 tablet_replica.cc:442] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:49.941630 2025 ts_tablet_manager.cc:1428] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 05:05:49.962649 2033 raft_consensus.cc:482] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:05:49.962935 2033 raft_consensus.cc:504] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.964939 2033 leader_election.cc:288] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c13c039c7aa4ec388714b1ec0d9a265 (127.0.97.1:37831), f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355)
I20260927 05:05:49.972702 2044 raft_consensus.cc:482] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:05:49.972914 2044 raft_consensus.cc:504] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.973822 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.965558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47071 (local address 127.0.97.3:46355)
0927 05:05:49.965795 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.965805 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.966712 (+ 907us) server_negotiation.cc:407] Connection header received
0927 05:05:49.966969 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.966975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.967061 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.967122 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:49.967862 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.968815 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.971168 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.972431 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.972559 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.972663 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.973382 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.973529 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 05:05:49.973803 2044 leader_election.cc:288] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a18149013f314502afe1474c26f2cee2 (127.0.97.2:37359), f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355)
I20260927 05:05:49.974570 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.965230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46355 (local address 127.0.97.2:47071)
0927 05:05:49.966613 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:49.966631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:49.966809 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:49.967302 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:49.967306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:49.967328 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:49.967699 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.967709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.968938 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.968945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.971003 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.971020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.972809 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.972816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.973142 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.973189 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 05:05:49.974002 (+ 813us) client_negotiation.cc:240] Negotiation successful
0927 05:05:49.974159 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"mutex_wait_us":28,"thread_start_us":1060,"threads_started":1}
I20260927 05:05:49.974910 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.965211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37831 (local address 127.0.97.2:39737)
0927 05:05:49.966353 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:49.966371 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:49.966571 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:49.966900 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:49.966904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:49.966928 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:49.967224 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.967232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.968585 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.968591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.970944 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:49.970960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.973164 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.973175 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:49.973439 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.973472 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:05:49.973847 (+ 375us) client_negotiation.cc:240] Negotiation successful
0927 05:05:49.974357 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":33,"thread_start_us":794,"threads_started":1}
I20260927 05:05:49.975178 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33f560b83f477f92ca45efd0442cca" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18149013f314502afe1474c26f2cee2" is_pre_election: true
I20260927 05:05:49.975440 1817 raft_consensus.cc:2454] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 0.
I20260927 05:05:49.975473 1961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33f560b83f477f92ca45efd0442cca" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" is_pre_election: true
I20260927 05:05:49.975716 1961 raft_consensus.cc:2454] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 0.
I20260927 05:05:49.975925 1608 leader_election.cc:302] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c13c039c7aa4ec388714b1ec0d9a265, a18149013f314502afe1474c26f2cee2; no voters:
I20260927 05:05:49.975983 1960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33f560b83f477f92ca45efd0442cca" candidate_uuid: "a18149013f314502afe1474c26f2cee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" is_pre_election: true
I20260927 05:05:49.976276 2044 raft_consensus.cc:2781] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:05:49.976275 1960 raft_consensus.cc:2454] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a18149013f314502afe1474c26f2cee2 in term 0.
I20260927 05:05:49.976375 2044 raft_consensus.cc:482] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:05:49.976483 2044 raft_consensus.cc:3037] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:49.976843 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:49.965620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39737 (local address 127.0.97.1:37831)
0927 05:05:49.965793 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:49.965799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:49.966461 (+ 662us) server_negotiation.cc:407] Connection header received
0927 05:05:49.966722 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:49.966726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:49.966782 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:49.966874 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:49.967397 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.968453 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.971125 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:49.972546 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:49.972703 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:49.972813 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:49.973636 (+ 823us) server_negotiation.cc:299] Negotiation successful
0927 05:05:49.973781 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:05:49.976827 1750 leader_election.cc:302] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a18149013f314502afe1474c26f2cee2, f3e9559d45ee4e06a8bbcc04d227f47a; no voters:
I20260927 05:05:49.977277 2033 raft_consensus.cc:2781] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:05:49.977382 2033 raft_consensus.cc:482] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:05:49.977470 2033 raft_consensus.cc:3037] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:49.977742 1677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33f560b83f477f92ca45efd0442cca" candidate_uuid: "a18149013f314502afe1474c26f2cee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" is_pre_election: true
I20260927 05:05:49.978945 2044 raft_consensus.cc:504] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.979370 1677 raft_consensus.cc:2379] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a18149013f314502afe1474c26f2cee2 in current term 1: Already voted for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in this term.
I20260927 05:05:49.980017 1817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33f560b83f477f92ca45efd0442cca" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a18149013f314502afe1474c26f2cee2"
I20260927 05:05:49.980535 2033 raft_consensus.cc:504] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.980860 1960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33f560b83f477f92ca45efd0442cca" candidate_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a"
I20260927 05:05:49.981053 1960 raft_consensus.cc:3037] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:49.981087 1817 raft_consensus.cc:2379] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c13c039c7aa4ec388714b1ec0d9a265 in current term 1: Already voted for candidate a18149013f314502afe1474c26f2cee2 in this term.
I20260927 05:05:49.981539 2044 leader_election.cc:288] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 election: Requested vote from peers a18149013f314502afe1474c26f2cee2 (127.0.97.2:37359), f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355)
I20260927 05:05:49.981998 1961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33f560b83f477f92ca45efd0442cca" candidate_uuid: "a18149013f314502afe1474c26f2cee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a"
I20260927 05:05:49.982692 2033 leader_election.cc:288] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [CANDIDATE]: Term 1 election: Requested vote from peers 7c13c039c7aa4ec388714b1ec0d9a265 (127.0.97.1:37831), f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355)
I20260927 05:05:49.982645 1677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33f560b83f477f92ca45efd0442cca" candidate_uuid: "a18149013f314502afe1474c26f2cee2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c13c039c7aa4ec388714b1ec0d9a265"
I20260927 05:05:49.982872 1677 raft_consensus.cc:2379] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a18149013f314502afe1474c26f2cee2 in current term 1: Already voted for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in this term.
I20260927 05:05:49.983284 1748 leader_election.cc:302] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a18149013f314502afe1474c26f2cee2; no voters: 7c13c039c7aa4ec388714b1ec0d9a265, f3e9559d45ee4e06a8bbcc04d227f47a
I20260927 05:05:49.983703 2033 raft_consensus.cc:2726] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 05:05:49.984391 1960 raft_consensus.cc:2454] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c13c039c7aa4ec388714b1ec0d9a265 in term 1.
I20260927 05:05:49.984812 1610 leader_election.cc:302] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c13c039c7aa4ec388714b1ec0d9a265, f3e9559d45ee4e06a8bbcc04d227f47a; no voters: a18149013f314502afe1474c26f2cee2
I20260927 05:05:49.985105 2044 raft_consensus.cc:2781] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:49.985321 2044 raft_consensus.cc:686] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 LEADER]: Becoming Leader. State: Replica: 7c13c039c7aa4ec388714b1ec0d9a265, State: Running, Role: LEADER
I20260927 05:05:49.985627 2044 consensus_queue.cc:237] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } }
I20260927 05:05:49.987388 2032 tablet_replica.cc:439] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: TxnStatusTablet state changed. Reason: New leader 7c13c039c7aa4ec388714b1ec0d9a265. Latest consensus state: current_term: 1 leader_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } }
I20260927 05:05:49.987694 2032 tablet_replica.cc:442] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:05:49.989089 1518 catalog_manager.cc:5495] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c13c039c7aa4ec388714b1ec0d9a265 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } health_report { overall_health: UNKNOWN } } }
I20260927 05:05:49.996486 2069 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 05:05:49.996748 2069 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 05:05:50.013550 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:50.008046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37831 (local address 127.0.0.1:44978)
0927 05:05:50.008281 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:50.008298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:50.008405 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:50.008918 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:50.008922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:50.008935 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:05:50.009114 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:50.009122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.010542 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.010550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:50.011597 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:50.011608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.012370 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.012375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:50.012538 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:50.012597 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:05:50.012999 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:05:50.013028 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 05:05:50.013114 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 05:05:50.013261 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 05:05:50.013520 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:50.008160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.0.97.1:37831)
0927 05:05:50.008643 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:50.008647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:50.008661 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:05:50.008720 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:50.008724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:50.008786 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:50.008891 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:50.009247 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.010405 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.011737 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.012223 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.012359 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:50.012796 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:50.012890 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:50.012983 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:50.013147 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 05:05:50.013220 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368}
I20260927 05:05:50.017052 1960 raft_consensus.cc:1261] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a [term 1 FOLLOWER]: Refusing update from remote peer 7c13c039c7aa4ec388714b1ec0d9a265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:50.017055 1817 raft_consensus.cc:1261] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Refusing update from remote peer 7c13c039c7aa4ec388714b1ec0d9a265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:50.017678 2032 consensus_queue.cc:1035] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:50.017989 2044 consensus_queue.cc:1035] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:50.022159 2031 tablet_replica.cc:439] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: TxnStatusTablet state changed. Reason: New leader 7c13c039c7aa4ec388714b1ec0d9a265. Latest consensus state: current_term: 1 leader_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } }
I20260927 05:05:50.022444 2031 tablet_replica.cc:442] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:50.023854 2036 tablet_replica.cc:439] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } }
I20260927 05:05:50.024245 2036 tablet_replica.cc:442] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:05:50.024963 2040 mvcc.cc:205] Tried to move back new op lower bound from 7333828812863770624 to 7333828812743274496. Current Snapshot: MvccSnapshot[applied={T|T < 7333828812863770624}]
I20260927 05:05:50.027012 2039 mvcc.cc:205] Tried to move back new op lower bound from 7333828812863770624 to 7333828812743274496. Current Snapshot: MvccSnapshot[applied={T|T < 7333828812863770624 or (T in {7333828812863770624})}]
I20260927 05:05:50.028826 2031 tablet_replica.cc:439] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } }
I20260927 05:05:50.029062 2031 tablet_replica.cc:442] T de33f560b83f477f92ca45efd0442cca P f3e9559d45ee4e06a8bbcc04d227f47a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:50.031594 2033 tablet_replica.cc:439] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: TxnStatusTablet state changed. Reason: New leader 7c13c039c7aa4ec388714b1ec0d9a265. Latest consensus state: current_term: 1 leader_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } }
I20260927 05:05:50.031888 2033 tablet_replica.cc:442] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:50.034780 2044 tablet_replica.cc:439] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } }
I20260927 05:05:50.035104 2044 tablet_replica.cc:442] T de33f560b83f477f92ca45efd0442cca P 7c13c039c7aa4ec388714b1ec0d9a265: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:05:50.035902 2033 tablet_replica.cc:439] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c13c039c7aa4ec388714b1ec0d9a265" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37831 } } peers { permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 } } peers { permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 } } }
I20260927 05:05:50.036217 2033 tablet_replica.cc:442] T de33f560b83f477f92ca45efd0442cca P a18149013f314502afe1474c26f2cee2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:50.050437 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:50.044108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45905 (local address 127.0.97.1:37831)
0927 05:05:50.044266 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:50.044270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:50.044923 (+ 653us) server_negotiation.cc:407] Connection header received
0927 05:05:50.045142 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:50.045147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:50.045228 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:50.045333 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:50.045853 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.046703 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.048522 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.049001 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.049145 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:50.049575 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:50.049700 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:50.049817 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:50.049980 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 05:05:50.050067 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":47}
I20260927 05:05:50.050531 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:50.044173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37831 (local address 127.0.97.1:45905)
0927 05:05:50.044816 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:50.044830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:50.045015 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:50.045344 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:50.045348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:50.045357 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:05:50.045706 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:50.045714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.046833 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.046838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:50.048377 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:50.048389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.049188 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.049196 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:50.049320 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:50.049377 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:05:50.049830 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:05:50.049846 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 05:05:50.049964 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 05:05:50.050120 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":380,"threads_started":1}
I20260927 05:05:50.062409 1817 raft_consensus.cc:1261] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Refusing update from remote peer 7c13c039c7aa4ec388714b1ec0d9a265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:50.063221 1960 raft_consensus.cc:1261] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a [term 1 FOLLOWER]: Refusing update from remote peer 7c13c039c7aa4ec388714b1ec0d9a265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:50.063522 2044 consensus_queue.cc:1035] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a18149013f314502afe1474c26f2cee2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:50.063906 2032 consensus_queue.cc:1035] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3e9559d45ee4e06a8bbcc04d227f47a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:50.073273 2041 mvcc.cc:205] Tried to move back new op lower bound from 7333828813048311808 to 7333828812104204288. Current Snapshot: MvccSnapshot[applied={T|T < 7333828813048311808}]
I20260927 05:05:50.141790 1518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:50.142112 1518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:50.146130 1518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 149fef5995694f15a8349f4ce4fc219a: Sending DeleteTablet for 3 replicas of tablet 62342c9235d84ab0bc50bf688a9f71c2
I20260927 05:05:50.147383 1657 tablet_service.cc:1514] Processing DeleteTablet for tablet 62342c9235d84ab0bc50bf688a9f71c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44956
I20260927 05:05:50.147356 1796 tablet_service.cc:1514] Processing DeleteTablet for tablet 62342c9235d84ab0bc50bf688a9f71c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46510
I20260927 05:05:50.147390 1936 tablet_service.cc:1514] Processing DeleteTablet for tablet 62342c9235d84ab0bc50bf688a9f71c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260927 05:05:50.147672 2049 tablet_replica.cc:330] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a: stopping tablet replica
I20260927 05:05:50.147667 2050 tablet_replica.cc:330] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265: stopping tablet replica
I20260927 05:05:50.147672 2051 tablet_replica.cc:330] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2: stopping tablet replica
I20260927 05:05:50.147869 2050 raft_consensus.cc:2229] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:05:50.147868 2051 raft_consensus.cc:2229] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:50.148100 2051 raft_consensus.cc:2258] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:50.147874 2049 raft_consensus.cc:2229] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:50.148222 2050 raft_consensus.cc:2258] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:50.148362 2049 raft_consensus.cc:2258] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:50.149629 2050 ts_tablet_manager.cc:1905] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:50.150664 2049 ts_tablet_manager.cc:1905] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:50.150844 2051 ts_tablet_manager.cc:1905] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:50.159041 2049 ts_tablet_manager.cc:1918] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:05:50.159569 2050 ts_tablet_manager.cc:1918] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:05:50.159729 2051 ts_tablet_manager.cc:1918] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:05:50.159790 2050 log.cc:1196] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/62342c9235d84ab0bc50bf688a9f71c2
I20260927 05:05:50.159899 2051 log.cc:1196] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/62342c9235d84ab0bc50bf688a9f71c2
I20260927 05:05:50.160406 2051 ts_tablet_manager.cc:1939] T 62342c9235d84ab0bc50bf688a9f71c2 P a18149013f314502afe1474c26f2cee2: Deleting consensus metadata
I20260927 05:05:50.161106 2049 log.cc:1196] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/62342c9235d84ab0bc50bf688a9f71c2
I20260927 05:05:50.161554 2049 ts_tablet_manager.cc:1939] T 62342c9235d84ab0bc50bf688a9f71c2 P f3e9559d45ee4e06a8bbcc04d227f47a: Deleting consensus metadata
I20260927 05:05:50.161803 1505 catalog_manager.cc:4841] TS a18149013f314502afe1474c26f2cee2 (127.0.97.2:37359): tablet 62342c9235d84ab0bc50bf688a9f71c2 (table test-table [id=6099a48649094d95b8b5dcf3347eb14a]) successfully deleted
I20260927 05:05:50.162392 2050 ts_tablet_manager.cc:1939] T 62342c9235d84ab0bc50bf688a9f71c2 P 7c13c039c7aa4ec388714b1ec0d9a265: Deleting consensus metadata
I20260927 05:05:50.162730 1507 catalog_manager.cc:4841] TS f3e9559d45ee4e06a8bbcc04d227f47a (127.0.97.3:46355): tablet 62342c9235d84ab0bc50bf688a9f71c2 (table test-table [id=6099a48649094d95b8b5dcf3347eb14a]) successfully deleted
I20260927 05:05:50.163892 1505 catalog_manager.cc:4841] TS 7c13c039c7aa4ec388714b1ec0d9a265 (127.0.97.1:37831): tablet 62342c9235d84ab0bc50bf688a9f71c2 (table test-table [id=6099a48649094d95b8b5dcf3347eb14a]) successfully deleted
Sep 27 05:05:50 dist-test-slave-hgs0 krb5kdc[1475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:05:50.181092 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:50.159778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.0.97.1:37831)
0927 05:05:50.160227 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:50.160236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:50.164329 (+ 4093us) server_negotiation.cc:407] Connection header received
0927 05:05:50.164673 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:50.164679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:50.164764 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:50.164885 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:50.166403 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.167201 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.168656 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.169115 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.169285 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:50.174128 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:50.174142 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:50.174158 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:50.174196 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:50.177829 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:50.178659 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:50.178668 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:50.178671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:50.178824 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:50.179281 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:50.179285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:50.179287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:50.179678 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:50.180131 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:50.180421 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 05:05:50.180728 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260927 05:05:50.181864 1722 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44988
I20260927 05:05:50.201833 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:50.185531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.0.97.62:32845)
0927 05:05:50.185814 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:50.185825 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:50.186317 (+ 492us) server_negotiation.cc:407] Connection header received
0927 05:05:50.186645 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:50.186654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:50.186781 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:50.186930 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:50.188582 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.189722 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.191047 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.191701 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.191853 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:50.194638 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:50.194645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:50.194651 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:50.194697 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:50.198256 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:50.199050 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:50.199056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:50.199060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:50.199220 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:50.200162 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:50.200168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:50.200171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:50.200555 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:50.200843 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:50.201232 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 05:05:50.201490 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 05:05:50.202491 1518 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42066
I20260927 05:05:50.204974 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 1591
I20260927 05:05:50.224364 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:50.205855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.0.97.1:37831)
0927 05:05:50.206583 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:50.206593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:50.208462 (+ 1869us) server_negotiation.cc:407] Connection header received
0927 05:05:50.208666 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:50.208670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:50.208781 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:50.208872 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:50.210740 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.211640 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.213547 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.214055 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.214185 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:50.217292 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:50.217297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:50.217301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:50.217339 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:50.221086 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:50.222046 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:50.222052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:50.222056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:50.222261 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:50.222711 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:50.222716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:50.222720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:50.223041 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:50.223313 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:50.223674 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 05:05:50.223974 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525}
W20260927 05:05:50.224932 1722 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44992
W20260927 05:05:50.225612 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 1591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:05:50.225855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 1591
I20260927 05:05:50.250708 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 1731
Sep 27 05:05:50 dist-test-slave-hgs0 krb5kdc[1475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:05:50.266958 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:50.251554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.0.97.2:37359)
0927 05:05:50.252171 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:50.252175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:50.252194 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:05:50.252275 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:50.252284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:50.252358 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:50.252483 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:50.253737 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.254495 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.255858 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.256403 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.256523 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:50.260943 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:50.260959 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:50.260976 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:50.261015 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:50.264166 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:50.264974 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:50.264983 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:50.264985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:50.265124 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:50.265487 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:50.265490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:50.265492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:50.265844 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:50.266085 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:50.266346 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 05:05:50.266571 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":304,"threads_started":1}
W20260927 05:05:50.267731 1860 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46512
W20260927 05:05:50.268432 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 1731: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:05:50.268630 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 1731
I20260927 05:05:50.285754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 1871
Sep 27 05:05:50 dist-test-slave-hgs0 krb5kdc[1475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485549, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:05:50.301093 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:50.286621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37606 (local address 127.0.97.3:46355)
0927 05:05:50.286807 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:50.286812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:50.286831 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:05:50.286889 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:50.286893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:50.286966 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:50.287065 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:50.288344 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.289148 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.290311 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:50.290754 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:50.290857 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:50.295285 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:50.295302 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:50.295319 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:50.295357 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:50.298255 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:50.299056 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:50.299064 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:50.299067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:50.299244 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:50.299660 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:50.299663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:50.299665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:50.300141 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:50.300382 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:50.300618 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 05:05:50.300794 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 05:05:50.301869 2001 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37606
W20260927 05:05:50.302562 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 1871: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:05:50.302702 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 1871
I20260927 05:05:50.320899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 1488
I20260927 05:05:50.322163 1560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:05:50.437623 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 1488
2026-09-27T05:05:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4750 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:05:50 dist-test-slave-hgs0 krb5kdc[2089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:05:50 dist-test-slave-hgs0 krb5kdc[2089](info): set up 2 sockets
Sep 27 05:05:50 dist-test-slave-hgs0 krb5kdc[2089](info): commencing operation
krb5kdc: starting...
W20260927 05:05:51.556233 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:05:51 dist-test-slave-hgs0 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:05:51.764272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37061
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:52.127345 2102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:52.127820 2102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:52.128053 2102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:52.137181 2102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:05:52.137377 2102 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 05:05:52.137499 2102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:52.137626 2102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:52.137737 2102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:05:52.137856 2102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:05:52.149013 2102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37061
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:52.150465 2102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:52.152015 2102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:52.165510 2108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:52.165584 2107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:52.165756 2110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:52.166625 2102 server_base.cc:1034] running on GCE node
I20260927 05:05:52.167567 2102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:52.169515 2102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:52.170835 2102 hybrid_clock.cc:648] HybridClock initialized: now 1790485552170748 us; error 56 us; skew 500 ppm
I20260927 05:05:52.171252 2102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:52 dist-test-slave-hgs0 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:52.174885 2102 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:05:52.176808 2102 webserver.cc:460] Webserver started at http://127.0.97.62:37723/ using document root <none> and password file <none>
I20260927 05:05:52.177645 2102 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:52.177749 2102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:52.178041 2102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:52.181660 2102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "76469c560eb64222925acbc9a0052276"
format_stamp: "Formatted at 2026-09-27 05:05:52 on dist-test-slave-hgs0"
server_key: "14d48466a14133a18107704fc181a6ac"
server_key_iv: "bd74d84d14d7977481321966e0cfa7c7"
server_key_version: "encryptionkey@0"
I20260927 05:05:52.182582 2102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "76469c560eb64222925acbc9a0052276"
format_stamp: "Formatted at 2026-09-27 05:05:52 on dist-test-slave-hgs0"
server_key: "14d48466a14133a18107704fc181a6ac"
server_key_iv: "bd74d84d14d7977481321966e0cfa7c7"
server_key_version: "encryptionkey@0"
I20260927 05:05:52.189255 2102 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 05:05:52.193512 2117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:52.195116 2102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:05:52.195330 2102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "76469c560eb64222925acbc9a0052276"
format_stamp: "Formatted at 2026-09-27 05:05:52 on dist-test-slave-hgs0"
server_key: "14d48466a14133a18107704fc181a6ac"
server_key_iv: "bd74d84d14d7977481321966e0cfa7c7"
server_key_version: "encryptionkey@0"
I20260927 05:05:52.195612 2102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:52.209139 2102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:52.214573 2102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:52.214927 2102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:52.241588 2102 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37061
I20260927 05:05:52.241609 2178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37061 every 8 connection(s)
I20260927 05:05:52.243229 2102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:05:52.244738 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 2102
I20260927 05:05:52.244961 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:05:52.245782 388 external_mini_cluster.cc:1428] Setting key 3efeae4c8b6b198bab2d5a65ebab8c86
I20260927 05:05:52.249105 2179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 05:05:52 dist-test-slave-hgs0 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:52.265290 2179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276: Bootstrap starting.
I20260927 05:05:52.269866 2179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:52.271274 2179 log.cc:824] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:52.274510 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:52.248330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.0.97.62:37061)
0927 05:05:52.249837 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:52.249851 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:52.249900 (+ 49us) server_negotiation.cc:407] Connection header received
0927 05:05:52.251843 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:52.251875 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:52.252477 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:52.252903 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:52.255102 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:52.256980 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:52.259159 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:52.260079 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:52.260238 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:52.265294 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:52.265308 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:52.265332 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:52.265384 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:52.269337 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:52.270289 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:52.270305 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:52.270317 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:52.270537 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:52.270984 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:52.270991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:52.270998 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:52.271759 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:52.272141 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:52.272653 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 05:05:52.273155 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1250,"thread_start_us":992,"threads_started":1}
I20260927 05:05:52.275038 2179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276: No bootstrap required, opened a new log
I20260927 05:05:52.283725 2179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76469c560eb64222925acbc9a0052276" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37061 } }
I20260927 05:05:52.284224 2179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:52.284303 2179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76469c560eb64222925acbc9a0052276, State: Initialized, Role: FOLLOWER
I20260927 05:05:52.284869 2179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76469c560eb64222925acbc9a0052276" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37061 } }
I20260927 05:05:52.285084 2179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:05:52.285180 2179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:05:52.285346 2179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:52.287740 2179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76469c560eb64222925acbc9a0052276" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37061 } }
I20260927 05:05:52.288545 2179 leader_election.cc:302] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76469c560eb64222925acbc9a0052276; no voters:
I20260927 05:05:52.289315 2179 leader_election.cc:288] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:05:52.289444 2184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:52.290333 2184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [term 1 LEADER]: Becoming Leader. State: Replica: 76469c560eb64222925acbc9a0052276, State: Running, Role: LEADER
I20260927 05:05:52.290957 2184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76469c560eb64222925acbc9a0052276" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37061 } }
I20260927 05:05:52.292137 2179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:05:52.294620 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76469c560eb64222925acbc9a0052276. Latest consensus state: current_term: 1 leader_uuid: "76469c560eb64222925acbc9a0052276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76469c560eb64222925acbc9a0052276" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37061 } } }
I20260927 05:05:52.294675 2185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76469c560eb64222925acbc9a0052276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76469c560eb64222925acbc9a0052276" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37061 } } }
I20260927 05:05:52.295219 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [sys.catalog]: This master's current role is: LEADER
I20260927 05:05:52.297981 2185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276 [sys.catalog]: This master's current role is: LEADER
I20260927 05:05:52.300211 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:05:52.306656 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:05:52.317624 2190 catalog_manager.cc:1347] Generated new cluster ID: 296d1c3d191946738c5f151e7d667959
I20260927 05:05:52.317793 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:05:52.330567 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:05:52.332024 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:05:52.341079 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276: Generated new TSK 0
I20260927 05:05:52.342067 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:05:52.393090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37061
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:52.754637 2205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:52.755035 2205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:52.755173 2205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:52.763537 2205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:52.763671 2205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:52.763759 2205 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 05:05:52.763811 2205 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:05:52.763883 2205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:05:52.774960 2205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:52.776505 2205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:52.777992 2205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:52.792853 2210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:52.792831 2213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:52.792790 2211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:52.793985 2205 server_base.cc:1034] running on GCE node
I20260927 05:05:52.794749 2205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:52.795826 2205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:52.797101 2205 hybrid_clock.cc:648] HybridClock initialized: now 1790485552797009 us; error 71 us; skew 500 ppm
I20260927 05:05:52.797551 2205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:52 dist-test-slave-hgs0 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:52.801868 2205 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:05:52.803633 2205 webserver.cc:460] Webserver started at http://127.0.97.1:41655/ using document root <none> and password file <none>
I20260927 05:05:52.804504 2205 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:52.804670 2205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:52.804967 2205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:52.808415 2205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "dff604d7d023462d8224a33e443c85e0"
format_stamp: "Formatted at 2026-09-27 05:05:52 on dist-test-slave-hgs0"
server_key: "9cbf8b1ab2c715238aff18d424345b19"
server_key_iv: "43b9328f44d9627ad3aa28bbb1088027"
server_key_version: "encryptionkey@0"
I20260927 05:05:52.809245 2205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dff604d7d023462d8224a33e443c85e0"
format_stamp: "Formatted at 2026-09-27 05:05:52 on dist-test-slave-hgs0"
server_key: "9cbf8b1ab2c715238aff18d424345b19"
server_key_iv: "43b9328f44d9627ad3aa28bbb1088027"
server_key_version: "encryptionkey@0"
I20260927 05:05:52.815197 2205 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 05:05:52.819375 2220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:52.821131 2205 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:05:52.821368 2205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "dff604d7d023462d8224a33e443c85e0"
format_stamp: "Formatted at 2026-09-27 05:05:52 on dist-test-slave-hgs0"
server_key: "9cbf8b1ab2c715238aff18d424345b19"
server_key_iv: "43b9328f44d9627ad3aa28bbb1088027"
server_key_version: "encryptionkey@0"
I20260927 05:05:52.821614 2205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:52.837929 2205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:52.842799 2205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:52.843118 2205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:52.845791 2205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:52.845881 2205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:52.846050 2205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:52.846139 2205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:05:52 dist-test-slave-hgs0 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:52.884338 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:52.846485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55889 (local address 127.0.97.62:37061)
0927 05:05:52.861803 (+ 15318us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:52.861808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:52.861833 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:05:52.861921 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:52.861928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:52.862037 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:52.862425 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:52.865238 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:52.866324 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:52.869365 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:52.869961 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:52.870121 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:52.876943 (+ 6822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:52.876955 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:52.876964 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:52.877017 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:52.880572 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:52.881640 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:52.881647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:52.881651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:52.881793 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:52.882309 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:52.882316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:52.882319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:52.882609 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:52.882781 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:52.883582 (+ 801us) server_negotiation.cc:299] Negotiation successful
0927 05:05:52.883820 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15181,"thread_start_us":15737,"threads_started":1}
I20260927 05:05:52.885715 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:52.847362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37061 (local address 127.0.97.1:55889)
0927 05:05:52.848507 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:52.848574 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:52.851718 (+ 3144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:52.862783 (+ 11065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:52.862795 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:52.863908 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:52.864990 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:52.865018 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:52.866500 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:52.866506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:52.869037 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:52.869074 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:52.870542 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:52.870554 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:52.871056 (+ 502us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:52.872223 (+ 1167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:52.872284 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:52.876468 (+ 4184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:52.880847 (+ 4379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:52.880860 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:52.880893 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:52.881480 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:52.881920 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:52.881926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:52.881931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:52.882152 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:52.882753 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:52.882762 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:52.883187 (+ 425us) client_negotiation.cc:769] Sending connection context
0927 05:05:52.883465 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 05:05:52.884093 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":512,"threads_started":1}
I20260927 05:05:52.893253 2205 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41525
I20260927 05:05:52.893375 2339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41525 every 8 connection(s)
I20260927 05:05:52.895299 2205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:05:52.897861 2341 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37061
I20260927 05:05:52.898185 2341 heartbeater.cc:464] Registering TS with master...
I20260927 05:05:52.899183 2341 heartbeater.cc:511] Master 127.0.97.62:37061 requested a full tablet report, sending...
I20260927 05:05:52.901777 2134 ts_manager.cc:194] Registered new tserver with Master: dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525)
I20260927 05:05:52.903401 2134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55889
I20260927 05:05:52.905023 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 2205
I20260927 05:05:52.905280 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:05:52.905918 388 external_mini_cluster.cc:1428] Setting key b695a13098ed3f09a0d532fe0e1e7133
I20260927 05:05:52.909509 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:52.894093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43143 (local address 127.0.97.62:37061)
0927 05:05:52.894303 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:52.894307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:52.895029 (+ 722us) server_negotiation.cc:407] Connection header received
0927 05:05:52.895267 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:52.895280 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:52.895352 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:52.895473 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:52.897039 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:52.898161 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:52.899913 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:52.900736 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:52.900907 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:52.902981 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:52.902987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:52.902990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:52.903023 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:52.906530 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:52.907458 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:52.907464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:52.907467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:52.907632 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:52.908060 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:52.908065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:52.908069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:52.908361 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:52.908532 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:52.908916 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 05:05:52.909156 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 05:05:52.909593 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:52.894110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37061 (local address 127.0.97.1:43143)
0927 05:05:52.894902 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:52.894920 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:52.895117 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:52.895505 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:52.895511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:52.896315 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:52.896814 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:52.896830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:52.898383 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:52.898387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:52.899727 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:52.899742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:52.900946 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:52.900953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:52.901171 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:52.901922 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:52.901970 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:52.902748 (+ 778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:52.906729 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:52.906734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:52.906738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:52.907283 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:52.907755 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:52.907760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:52.907764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:52.907900 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:52.908488 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:52.908494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:52.908673 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 05:05:52.908819 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 05:05:52.909152 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":451,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:05:52.952482 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37061
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:53.303107 2345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:53.303481 2345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:53.303678 2345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:53.311728 2345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:53.311909 2345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:53.312104 2345 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 05:05:53.312197 2345 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:05:53.312331 2345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:05:53.322908 2345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:53.324429 2345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:53.326056 2345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:53.340268 2351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:53.340216 2353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:53.340922 2350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:53.341616 2345 server_base.cc:1034] running on GCE node
I20260927 05:05:53.342365 2345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:53.343454 2345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:53.344702 2345 hybrid_clock.cc:648] HybridClock initialized: now 1790485553344626 us; error 67 us; skew 500 ppm
I20260927 05:05:53.345101 2345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:53 dist-test-slave-hgs0 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:53.354647 2345 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:05:53.356411 2345 webserver.cc:460] Webserver started at http://127.0.97.2:40941/ using document root <none> and password file <none>
I20260927 05:05:53.357187 2345 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:53.357347 2345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:53.357683 2345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:53.360860 2345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "350b0feacf8d4533b286a1de137550ad"
format_stamp: "Formatted at 2026-09-27 05:05:53 on dist-test-slave-hgs0"
server_key: "5891ebd14abbe63ca41f7b810105c8a4"
server_key_iv: "120469b6343579ecd1205767eda5a7cc"
server_key_version: "encryptionkey@0"
I20260927 05:05:53.361713 2345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "350b0feacf8d4533b286a1de137550ad"
format_stamp: "Formatted at 2026-09-27 05:05:53 on dist-test-slave-hgs0"
server_key: "5891ebd14abbe63ca41f7b810105c8a4"
server_key_iv: "120469b6343579ecd1205767eda5a7cc"
server_key_version: "encryptionkey@0"
I20260927 05:05:53.367512 2345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:05:53.371918 2360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:53.373834 2345 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:05:53.374060 2345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "350b0feacf8d4533b286a1de137550ad"
format_stamp: "Formatted at 2026-09-27 05:05:53 on dist-test-slave-hgs0"
server_key: "5891ebd14abbe63ca41f7b810105c8a4"
server_key_iv: "120469b6343579ecd1205767eda5a7cc"
server_key_version: "encryptionkey@0"
I20260927 05:05:53.374323 2345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:53.413483 2345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:53.418279 2345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:53.418603 2345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:53.421633 2345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:53.421769 2345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:53.421891 2345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:53.422003 2345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:05:53 dist-test-slave-hgs0 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:53.448283 2369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:53.422282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56835 (local address 127.0.97.62:37061)
0927 05:05:53.422941 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:53.422946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:53.424263 (+ 1317us) server_negotiation.cc:407] Connection header received
0927 05:05:53.426603 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:53.426609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:53.426719 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:53.426854 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:53.429588 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.430880 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.434980 (+ 4100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.435619 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.435792 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:53.441587 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:53.441593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:53.441596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:53.441634 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:53.444658 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:53.445757 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:53.445763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:53.445767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:53.445910 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:53.446378 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:53.446383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:53.446387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:53.446702 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:53.446862 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:53.447627 (+ 765us) server_negotiation.cc:299] Negotiation successful
0927 05:05:53.447852 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":448,"threads_started":1}
I20260927 05:05:53.450343 2370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:53.423013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37061 (local address 127.0.97.2:56835)
0927 05:05:53.424079 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:53.424131 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:53.426282 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:53.427193 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:53.427214 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:53.428186 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:53.429348 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:53.429390 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.431234 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.431244 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:53.434699 (+ 3455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:53.434727 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.436231 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.436242 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:53.436577 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:53.437682 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:53.437721 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:53.441289 (+ 3568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:53.444878 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:53.444885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:53.444901 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:53.445600 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:53.446027 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:53.446032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:53.446035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:53.446245 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:53.446836 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:53.446844 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:53.447242 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 05:05:53.447515 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 05:05:53.448180 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":432,"threads_started":1}
I20260927 05:05:53.470908 2345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43411
I20260927 05:05:53.470908 2480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43411 every 8 connection(s)
I20260927 05:05:53.472779 2345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:05:53.473158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 2345
I20260927 05:05:53.473354 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:05:53.474035 388 external_mini_cluster.cc:1428] Setting key 72bbc1fb6091cc168e3551ab2b2fe28e
I20260927 05:05:53.476379 2481 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37061
I20260927 05:05:53.476744 2481 heartbeater.cc:464] Registering TS with master...
I20260927 05:05:53.478574 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:53.460411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37061 (local address 127.0.97.2:52853)
0927 05:05:53.461138 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:53.461167 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:53.461353 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:53.461839 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:53.461847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:53.462400 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:53.462711 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:53.462721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.464415 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.464426 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:53.465866 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:53.465886 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.467093 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.467103 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:53.467335 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:53.468324 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:53.468360 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:53.469264 (+ 904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:53.473782 (+ 4518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:53.473791 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:53.473798 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:53.474615 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:53.475238 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:53.475244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:53.475248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:53.475445 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:53.476631 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:53.476640 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:53.476956 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 05:05:53.477143 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 05:05:53.477721 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":397,"threads_started":1}
I20260927 05:05:53.477867 2481 heartbeater.cc:511] Master 127.0.97.62:37061 requested a full tablet report, sending...
I20260927 05:05:53.479796 2369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:53.460411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52853 (local address 127.0.97.62:37061)
0927 05:05:53.460636 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:53.460642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:53.461284 (+ 642us) server_negotiation.cc:407] Connection header received
0927 05:05:53.461552 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:53.461560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:53.461657 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:53.461782 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:53.462877 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.464218 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.466044 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.466846 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.467004 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:53.469531 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:53.469541 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:53.469561 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:53.469613 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:53.473580 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:53.474885 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:53.474893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:53.474900 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:53.475083 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:53.475612 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:53.475620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:53.475626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:53.476339 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:53.478112 (+ 1773us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:53.478313 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 05:05:53.479173 (+ 860us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":66}
I20260927 05:05:53.481055 2133 ts_manager.cc:194] Registered new tserver with Master: 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411)
I20260927 05:05:53.482218 2133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56835
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:05:53.524775 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37061
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:05:53.882133 2485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:05:53.882521 2485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:05:53.882731 2485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:05:53.892642 2485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:05:53.892819 2485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:05:53.893007 2485 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 05:05:53.893115 2485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:05:53.893261 2485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:05:53.907665 2485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:05:53.907753 2341 heartbeater.cc:503] Master 127.0.97.62:37061 was elected leader, sending a full tablet report...
I20260927 05:05:53.909416 2485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:05:53.911203 2485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:05:53.927524 2491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:53.927435 2493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:05:53.927489 2490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:05:53.928498 2485 server_base.cc:1034] running on GCE node
I20260927 05:05:53.929234 2485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:05:53.930348 2485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:05:53.931610 2485 hybrid_clock.cc:648] HybridClock initialized: now 1790485553931559 us; error 38 us; skew 500 ppm
I20260927 05:05:53.932047 2485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:05:53 dist-test-slave-hgs0 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:05:53.935508 2485 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:05:53.937316 2485 webserver.cc:460] Webserver started at http://127.0.97.3:43039/ using document root <none> and password file <none>
I20260927 05:05:53.938038 2485 fs_manager.cc:359] Metadata directory not provided
I20260927 05:05:53.938167 2485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:05:53.938441 2485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:05:53.941660 2485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "19bb670db53f43f09b3b0ac82dc4dd5f"
format_stamp: "Formatted at 2026-09-27 05:05:53 on dist-test-slave-hgs0"
server_key: "1ccb739b9d1b91cf9717bbcad2403981"
server_key_iv: "bcee7cce20afe9b72634c175f7e29abf"
server_key_version: "encryptionkey@0"
I20260927 05:05:53.942451 2485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "19bb670db53f43f09b3b0ac82dc4dd5f"
format_stamp: "Formatted at 2026-09-27 05:05:53 on dist-test-slave-hgs0"
server_key: "1ccb739b9d1b91cf9717bbcad2403981"
server_key_iv: "bcee7cce20afe9b72634c175f7e29abf"
server_key_version: "encryptionkey@0"
I20260927 05:05:53.948309 2485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 05:05:53.952420 2500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:53.954265 2485 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:05:53.954427 2485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "19bb670db53f43f09b3b0ac82dc4dd5f"
format_stamp: "Formatted at 2026-09-27 05:05:53 on dist-test-slave-hgs0"
server_key: "1ccb739b9d1b91cf9717bbcad2403981"
server_key_iv: "bcee7cce20afe9b72634c175f7e29abf"
server_key_version: "encryptionkey@0"
I20260927 05:05:53.954630 2485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:05:53.973444 2485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:05:53.978458 2485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:05:53.978794 2485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:05:53.982108 2485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:05:53.982239 2485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:05:53.982414 2485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:05:53.982529 2485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:05:53 dist-test-slave-hgs0 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:54.012140 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:53.982770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56833 (local address 127.0.97.62:37061)
0927 05:05:53.985839 (+ 3069us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:53.985848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:53.985876 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:05:53.988216 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:53.988228 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:53.988349 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:53.988453 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:53.991862 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.993253 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.997085 (+ 3832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.997712 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.997874 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.003862 (+ 5988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:54.003871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:54.003878 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:54.004055 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:54.008182 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:54.009394 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:54.009400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:54.009404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:54.009570 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:54.010091 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:54.010097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:54.010101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:54.010402 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:54.010601 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.011421 (+ 820us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.011681 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2912,"thread_start_us":2785,"threads_started":1}
I20260927 05:05:54.013412 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:53.983620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37061 (local address 127.0.97.3:56833)
0927 05:05:53.985005 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:53.985101 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:53.987688 (+ 2587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:53.989270 (+ 1582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:53.989288 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:53.990358 (+ 1070us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:53.991614 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:53.991640 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.993474 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.993482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:53.996819 (+ 3337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:53.996850 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:53.998098 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:53.998105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:53.998315 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:53.999358 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:53.999401 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:54.003492 (+ 4091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:54.008443 (+ 4951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:54.008452 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:54.008469 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:54.009222 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:54.009706 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:54.009711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:54.009717 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:54.009945 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:54.010551 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:54.010560 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:54.010981 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.011292 (+ 311us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.011904 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":485,"threads_started":1}
I20260927 05:05:54.039470 2485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43945
I20260927 05:05:54.039570 2620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43945 every 8 connection(s)
I20260927 05:05:54.040967 2485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:05:54.043385 2621 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37061
I20260927 05:05:54.043675 2621 heartbeater.cc:464] Registering TS with master...
I20260927 05:05:54.044629 2621 heartbeater.cc:511] Master 127.0.97.62:37061 requested a full tablet report, sending...
I20260927 05:05:54.045260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 2485
I20260927 05:05:54.045452 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:05:54.046063 388 external_mini_cluster.cc:1428] Setting key 36e159b1b731bbe5bd3d91e0f86a13ab
I20260927 05:05:54.046303 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.023912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56095 (local address 127.0.97.62:37061)
0927 05:05:54.024175 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.024182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.029395 (+ 5213us) server_negotiation.cc:407] Connection header received
0927 05:05:54.029652 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.029657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.029735 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.029842 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:54.031484 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.032471 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.035524 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.036390 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.036530 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.039595 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:54.039606 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:54.039613 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:54.039663 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:54.043035 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:54.043995 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:54.044002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:54.044012 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:54.044151 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:54.044573 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:54.044576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:54.044578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:54.045025 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:54.045183 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.045676 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.045917 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 05:05:54.046375 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.023765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37061 (local address 127.0.97.3:56095)
0927 05:05:54.029208 (+ 5443us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.029241 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.029489 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.029921 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.029930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.030742 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:54.031256 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.031277 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.032714 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.032729 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.035236 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.035268 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.036957 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.036964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.037223 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.037922 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:54.037951 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:54.039324 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:54.043192 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:54.043198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:54.043201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:54.043820 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:54.044293 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:54.044298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:54.044301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:54.044463 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:54.045191 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:54.045196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:54.045417 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.045572 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.045917 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5182,"thread_start_us":4896,"threads_started":1}
I20260927 05:05:54.047204 2134 ts_manager.cc:194] Registered new tserver with Master: 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945)
I20260927 05:05:54.048483 2134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56833
I20260927 05:05:54.050056 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 05:05:54 dist-test-slave-hgs0 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 05:05:54 dist-test-slave-hgs0 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:54.102089 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.087285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.0.97.62:37061)
0927 05:05:54.087528 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.087535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.087728 (+ 193us) server_negotiation.cc:407] Connection header received
0927 05:05:54.088028 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.088033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.088110 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.088260 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:54.089735 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.090466 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.091737 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.092287 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.092431 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.096665 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:54.096670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:54.096674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:54.096712 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:54.099325 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:54.100078 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:54.100083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:54.100087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:54.100230 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:54.100640 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:54.100647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:54.100650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:54.100940 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:54.101246 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.101507 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.101734 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:05:54.106498 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:05:54.113526 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:05:54.143313 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.128949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43411 (local address 127.0.0.1:53066)
0927 05:05:54.131604 (+ 2655us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.131647 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.131883 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.133237 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.133254 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.133291 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:54.133948 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.133961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.136665 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.136671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.140397 (+ 3726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.140412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.142107 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.142115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.142343 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.142435 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.142571 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.142825 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2430,"mutex_wait_us":56,"thread_start_us":3317,"threads_started":1}
I20260927 05:05:54.144608 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.129196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53066 (local address 127.0.97.2:43411)
0927 05:05:54.130232 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.130243 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.132502 (+ 2259us) server_negotiation.cc:407] Connection header received
0927 05:05:54.132666 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.132676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.132980 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.134224 (+ 1244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:54.134704 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.136465 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.140601 (+ 4136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.141865 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.143562 (+ 1697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.143878 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.143998 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.144215 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":756,"threads_started":1}
I20260927 05:05:54.145574 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.130412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.0.97.3:43945)
0927 05:05:54.131143 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.131155 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.132850 (+ 1695us) server_negotiation.cc:407] Connection header received
0927 05:05:54.132948 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.132956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.133243 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.133444 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:54.134915 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.136650 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.141803 (+ 5153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.143744 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.143909 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.144210 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.145003 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.145234 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":392,"threads_started":1}
I20260927 05:05:54.146762 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.129788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43945 (local address 127.0.0.1:46436)
0927 05:05:54.132583 (+ 2795us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.132602 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.132724 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.133914 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.133925 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.133953 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:54.134645 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.134662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.136823 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.136834 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.141561 (+ 4727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.141578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.144405 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.144413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.144717 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.144777 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.146005 (+ 1228us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.146294 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2639,"mutex_wait_us":8,"thread_start_us":1470,"threads_started":1}
I20260927 05:05:54.148442 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.133472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41525 (local address 127.0.0.1:34560)
0927 05:05:54.138786 (+ 5314us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.138804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.138929 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.139687 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.139693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.139731 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:54.140408 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.140426 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.142673 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.142679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.145140 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.145154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.147475 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.147483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.147695 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.147732 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.147836 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.148017 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5167,"mutex_wait_us":72,"thread_start_us":4981,"threads_started":1}
I20260927 05:05:54.149262 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.132963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.0.97.1:41525)
0927 05:05:54.138334 (+ 5371us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.138345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.138937 (+ 592us) server_negotiation.cc:407] Connection header received
0927 05:05:54.139093 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.139104 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.139427 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.139669 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:54.140698 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.142451 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.145344 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.146551 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.146744 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.146969 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.148766 (+ 1797us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.148970 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5182,"thread_start_us":5026,"threads_started":1}
I20260927 05:05:54.149626 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 424b81e6fe9a4500aca20b9d39d0188a (DEFAULT_TABLE table=test-table [id=42bc03e79cf5472e9a9ae10d93f245f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:54.151347 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424b81e6fe9a4500aca20b9d39d0188a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:54.152678 2271 tablet_service.cc:1467] Processing CreateTablet for tablet 424b81e6fe9a4500aca20b9d39d0188a (DEFAULT_TABLE table=test-table [id=42bc03e79cf5472e9a9ae10d93f245f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:54.152704 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 424b81e6fe9a4500aca20b9d39d0188a (DEFAULT_TABLE table=test-table [id=42bc03e79cf5472e9a9ae10d93f245f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:54.154206 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424b81e6fe9a4500aca20b9d39d0188a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:54.154251 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424b81e6fe9a4500aca20b9d39d0188a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:54.167273 2640 tablet_bootstrap.cc:492] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: Bootstrap starting.
I20260927 05:05:54.168545 2641 tablet_bootstrap.cc:492] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: Bootstrap starting.
I20260927 05:05:54.172380 2642 tablet_bootstrap.cc:492] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: Bootstrap starting.
I20260927 05:05:54.172823 2641 tablet_bootstrap.cc:654] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:54.173485 2640 tablet_bootstrap.cc:654] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:54.174401 2641 log.cc:824] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:54.174715 2640 log.cc:824] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:54.177035 2642 tablet_bootstrap.cc:654] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:54.178134 2640 tablet_bootstrap.cc:492] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: No bootstrap required, opened a new log
I20260927 05:05:54.178134 2641 tablet_bootstrap.cc:492] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: No bootstrap required, opened a new log
I20260927 05:05:54.178224 2642 log.cc:824] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: Log is configured to *not* fsync() on all Append() calls
I20260927 05:05:54.178495 2640 ts_tablet_manager.cc:1397] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 05:05:54.178494 2641 ts_tablet_manager.cc:1397] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s
I20260927 05:05:54.181684 2642 tablet_bootstrap.cc:492] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: No bootstrap required, opened a new log
I20260927 05:05:54.182127 2642 ts_tablet_manager.cc:1397] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260927 05:05:54.191264 2642 raft_consensus.cc:348] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.191103 2640 raft_consensus.cc:348] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.191103 2641 raft_consensus.cc:348] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.191699 2642 raft_consensus.cc:374] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:54.191699 2641 raft_consensus.cc:374] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:54.191699 2640 raft_consensus.cc:374] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:54.191807 2642 raft_consensus.cc:729] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bb670db53f43f09b3b0ac82dc4dd5f, State: Initialized, Role: FOLLOWER
I20260927 05:05:54.191812 2640 raft_consensus.cc:729] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff604d7d023462d8224a33e443c85e0, State: Initialized, Role: FOLLOWER
I20260927 05:05:54.191812 2641 raft_consensus.cc:729] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350b0feacf8d4533b286a1de137550ad, State: Initialized, Role: FOLLOWER
I20260927 05:05:54.192585 2642 consensus_queue.cc:260] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.192606 2640 consensus_queue.cc:260] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.192607 2641 consensus_queue.cc:260] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.194877 2641 ts_tablet_manager.cc:1428] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260927 05:05:54.195075 2640 ts_tablet_manager.cc:1428] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: Time spent starting tablet: real 0.016s user 0.013s sys 0.005s
I20260927 05:05:54.195910 2481 heartbeater.cc:503] Master 127.0.97.62:37061 was elected leader, sending a full tablet report...
I20260927 05:05:54.197919 2621 heartbeater.cc:503] Master 127.0.97.62:37061 was elected leader, sending a full tablet report...
I20260927 05:05:54.198657 2642 ts_tablet_manager.cc:1428] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260927 05:05:54.222775 2648 raft_consensus.cc:482] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:05:54.223207 2648 raft_consensus.cc:504] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.225426 2648 leader_election.cc:288] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411), dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525)
W20260927 05:05:54.232395 2482 tablet.cc:2369] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:05:54.232882 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.225851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34595 (local address 127.0.97.2:43411)
0927 05:05:54.226089 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.226098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.226123 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:05:54.226418 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.226425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.226521 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.226639 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:54.227305 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.228388 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.230618 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.231819 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.231990 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.232110 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.232440 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.232577 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 05:05:54.232928 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.225779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43411 (local address 127.0.97.3:34595)
0927 05:05:54.226027 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.226048 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.226215 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.226809 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.226816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.226860 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:54.227093 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.227104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.228564 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.228571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.230428 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.230443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.232006 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.232012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.232267 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.232309 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.232400 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.232675 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 05:05:54.234402 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.226232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41525 (local address 127.0.97.3:40861)
0927 05:05:54.227075 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.227090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.227234 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.227618 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.227625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.227650 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:54.228013 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.228024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.229527 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.229531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.231118 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.231131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.233758 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.233764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.233945 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.233979 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.234054 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.234145 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":3,"thread_start_us":502,"threads_started":1}
I20260927 05:05:54.234675 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.226420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40861 (local address 127.0.97.1:41525)
0927 05:05:54.226674 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.226681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.227202 (+ 521us) server_negotiation.cc:407] Connection header received
0927 05:05:54.227390 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.227394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.227480 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.227569 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:54.228185 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.229358 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.231368 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.233505 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.233667 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.233758 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.234125 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.234291 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:05:54.234501 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424b81e6fe9a4500aca20b9d39d0188a" candidate_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350b0feacf8d4533b286a1de137550ad" is_pre_election: true
I20260927 05:05:54.235165 2431 raft_consensus.cc:2454] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bb670db53f43f09b3b0ac82dc4dd5f in term 0.
I20260927 05:05:54.235775 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424b81e6fe9a4500aca20b9d39d0188a" candidate_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff604d7d023462d8224a33e443c85e0" is_pre_election: true
I20260927 05:05:54.236053 2501 leader_election.cc:302] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bb670db53f43f09b3b0ac82dc4dd5f, 350b0feacf8d4533b286a1de137550ad; no voters:
I20260927 05:05:54.236553 2648 raft_consensus.cc:2781] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:05:54.236553 2291 raft_consensus.cc:2454] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bb670db53f43f09b3b0ac82dc4dd5f in term 0.
I20260927 05:05:54.236749 2648 raft_consensus.cc:482] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:05:54.236891 2648 raft_consensus.cc:3037] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:54.239657 2648 raft_consensus.cc:504] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.240417 2648 leader_election.cc:288] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [CANDIDATE]: Term 1 election: Requested vote from peers 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411), dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525)
I20260927 05:05:54.241168 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424b81e6fe9a4500aca20b9d39d0188a" candidate_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350b0feacf8d4533b286a1de137550ad"
I20260927 05:05:54.241225 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424b81e6fe9a4500aca20b9d39d0188a" candidate_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff604d7d023462d8224a33e443c85e0"
I20260927 05:05:54.241415 2431 raft_consensus.cc:3037] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:54.241461 2291 raft_consensus.cc:3037] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:54.243921 2431 raft_consensus.cc:2454] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bb670db53f43f09b3b0ac82dc4dd5f in term 1.
I20260927 05:05:54.244540 2501 leader_election.cc:302] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bb670db53f43f09b3b0ac82dc4dd5f, 350b0feacf8d4533b286a1de137550ad; no voters:
I20260927 05:05:54.244894 2648 raft_consensus.cc:2781] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:54.245055 2291 raft_consensus.cc:2454] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bb670db53f43f09b3b0ac82dc4dd5f in term 1.
I20260927 05:05:54.245544 2648 raft_consensus.cc:686] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 LEADER]: Becoming Leader. State: Replica: 19bb670db53f43f09b3b0ac82dc4dd5f, State: Running, Role: LEADER
I20260927 05:05:54.246201 2648 consensus_queue.cc:237] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.251622 2133 catalog_manager.cc:5495] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } health_report { overall_health: HEALTHY } } }
W20260927 05:05:54.293213 2622 tablet.cc:2369] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 05:05:54 dist-test-slave-hgs0 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:54.308092 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.290960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37061 (local address 127.0.0.1:41752)
0927 05:05:54.291655 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.291673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.291822 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.292336 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.292340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.293253 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:05:54.293727 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.293739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.295067 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.295072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.296430 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.296445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.297476 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.297482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.297665 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.298483 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:05:54.298503 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:05:54.301421 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:05:54.305187 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:54.305202 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:54.305221 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:54.305717 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:54.306174 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:05:54.306180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:05:54.306183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:05:54.306456 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:05:54.307061 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:05:54.307071 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:05:54.307240 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.307384 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.307680 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":391,"threads_started":1}
I20260927 05:05:54.308372 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.290846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.0.97.62:37061)
0927 05:05:54.291097 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.291107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.291776 (+ 669us) server_negotiation.cc:407] Connection header received
0927 05:05:54.292042 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.292046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.292115 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.292221 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:54.293937 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.294877 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.296704 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.297222 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.297346 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.301751 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:54.301758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:54.301763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:54.301803 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:54.305008 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:54.305833 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:54.305837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:54.305839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:54.305956 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:54.306600 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:54.306604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:54.306607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:54.306920 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:54.307073 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.307481 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.307856 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":163,"server-negotiator.queue_time_us":83}
I20260927 05:05:54.313617 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41752:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 05:05:54.314786 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:05:54.321964 2271 tablet_service.cc:1467] Processing CreateTablet for tablet db72f2bb8d0e414b8fec22aff03c95aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b82acf9a9b614c31a86bd112bee85a08]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:05:54.321964 2411 tablet_service.cc:1467] Processing CreateTablet for tablet db72f2bb8d0e414b8fec22aff03c95aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b82acf9a9b614c31a86bd112bee85a08]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:05:54.322314 2551 tablet_service.cc:1467] Processing CreateTablet for tablet db72f2bb8d0e414b8fec22aff03c95aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b82acf9a9b614c31a86bd112bee85a08]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:05:54.323071 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db72f2bb8d0e414b8fec22aff03c95aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:54.323058 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db72f2bb8d0e414b8fec22aff03c95aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:54.323236 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db72f2bb8d0e414b8fec22aff03c95aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:54.330147 2642 tablet_bootstrap.cc:492] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: Bootstrap starting.
I20260927 05:05:54.330535 2640 tablet_bootstrap.cc:492] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: Bootstrap starting.
I20260927 05:05:54.330681 2641 tablet_bootstrap.cc:492] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: Bootstrap starting.
I20260927 05:05:54.332171 2642 tablet_bootstrap.cc:654] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:54.332265 2640 tablet_bootstrap.cc:654] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:54.332589 2641 tablet_bootstrap.cc:654] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:54.334720 2640 tablet_bootstrap.cc:492] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: No bootstrap required, opened a new log
I20260927 05:05:54.334901 2640 ts_tablet_manager.cc:1397] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 05:05:54.335366 2642 tablet_bootstrap.cc:492] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: No bootstrap required, opened a new log
I20260927 05:05:54.335536 2642 ts_tablet_manager.cc:1397] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 05:05:54.335680 2640 raft_consensus.cc:348] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.335983 2640 raft_consensus.cc:374] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:54.336097 2640 raft_consensus.cc:729] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff604d7d023462d8224a33e443c85e0, State: Initialized, Role: FOLLOWER
I20260927 05:05:54.336392 2640 consensus_queue.cc:260] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.336442 2642 raft_consensus.cc:348] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.336735 2642 raft_consensus.cc:374] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:54.336827 2642 raft_consensus.cc:729] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bb670db53f43f09b3b0ac82dc4dd5f, State: Initialized, Role: FOLLOWER
I20260927 05:05:54.336882 2646 tablet_replica.cc:439] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } } }
I20260927 05:05:54.337055 2642 consensus_queue.cc:260] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.337273 2646 tablet_replica.cc:442] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:54.337642 2642 ts_tablet_manager.cc:1428] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:05:54.338030 2648 tablet_replica.cc:439] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } } }
I20260927 05:05:54.338379 2648 tablet_replica.cc:442] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:54.339562 2641 tablet_bootstrap.cc:492] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: No bootstrap required, opened a new log
I20260927 05:05:54.339726 2641 ts_tablet_manager.cc:1397] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260927 05:05:54.339859 2640 ts_tablet_manager.cc:1428] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260927 05:05:54.340662 2641 raft_consensus.cc:348] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.340956 2641 raft_consensus.cc:374] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:54.341076 2641 raft_consensus.cc:729] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350b0feacf8d4533b286a1de137550ad, State: Initialized, Role: FOLLOWER
I20260927 05:05:54.341333 2641 consensus_queue.cc:260] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.341835 2647 tablet_replica.cc:439] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } } }
I20260927 05:05:54.342226 2647 tablet_replica.cc:442] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:54.343427 2641 ts_tablet_manager.cc:1428] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
W20260927 05:05:54.402456 2342 tablet.cc:2369] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:05:54.468052 2648 raft_consensus.cc:482] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:05:54.468294 2648 raft_consensus.cc:504] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.468989 2648 leader_election.cc:288] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525), 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411)
I20260927 05:05:54.469661 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db72f2bb8d0e414b8fec22aff03c95aa" candidate_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350b0feacf8d4533b286a1de137550ad" is_pre_election: true
I20260927 05:05:54.469656 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db72f2bb8d0e414b8fec22aff03c95aa" candidate_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff604d7d023462d8224a33e443c85e0" is_pre_election: true
I20260927 05:05:54.469925 2291 raft_consensus.cc:2454] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bb670db53f43f09b3b0ac82dc4dd5f in term 0.
I20260927 05:05:54.469941 2431 raft_consensus.cc:2454] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bb670db53f43f09b3b0ac82dc4dd5f in term 0.
I20260927 05:05:54.470372 2501 leader_election.cc:302] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bb670db53f43f09b3b0ac82dc4dd5f, dff604d7d023462d8224a33e443c85e0; no voters:
I20260927 05:05:54.470666 2648 raft_consensus.cc:2781] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:05:54.470783 2648 raft_consensus.cc:482] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:05:54.470901 2648 raft_consensus.cc:3037] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:54.473841 2648 raft_consensus.cc:504] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.474524 2648 leader_election.cc:288] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [CANDIDATE]: Term 1 election: Requested vote from peers dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525), 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411)
I20260927 05:05:54.475010 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db72f2bb8d0e414b8fec22aff03c95aa" candidate_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff604d7d023462d8224a33e443c85e0"
I20260927 05:05:54.475037 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db72f2bb8d0e414b8fec22aff03c95aa" candidate_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350b0feacf8d4533b286a1de137550ad"
I20260927 05:05:54.475274 2291 raft_consensus.cc:3037] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:54.475298 2431 raft_consensus.cc:3037] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:54.477623 2431 raft_consensus.cc:2454] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bb670db53f43f09b3b0ac82dc4dd5f in term 1.
I20260927 05:05:54.478119 2501 leader_election.cc:302] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bb670db53f43f09b3b0ac82dc4dd5f, 350b0feacf8d4533b286a1de137550ad; no voters:
I20260927 05:05:54.478248 2291 raft_consensus.cc:2454] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bb670db53f43f09b3b0ac82dc4dd5f in term 1.
I20260927 05:05:54.478456 2648 raft_consensus.cc:2781] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:54.478876 2648 raft_consensus.cc:686] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 LEADER]: Becoming Leader. State: Replica: 19bb670db53f43f09b3b0ac82dc4dd5f, State: Running, Role: LEADER
I20260927 05:05:54.479302 2648 consensus_queue.cc:237] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.481174 2650 tablet_replica.cc:439] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: TxnStatusTablet state changed. Reason: New leader 19bb670db53f43f09b3b0ac82dc4dd5f. Latest consensus state: current_term: 1 leader_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } } }
I20260927 05:05:54.481537 2650 tablet_replica.cc:442] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:05:54.482453 2662 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 05:05:54.482777 2662 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 05:05:54.482852 2133 catalog_manager.cc:5495] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } health_report { overall_health: HEALTHY } } }
I20260927 05:05:54.533254 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.526651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43945 (local address 127.0.0.1:46450)
0927 05:05:54.526954 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.526974 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.527116 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.527631 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.527635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.527645 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:05:54.527805 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.527812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.529297 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.529301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.530321 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.530332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.531399 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.531404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.531589 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.531649 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:05:54.532789 (+ 1140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:05:54.532811 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.532886 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.533017 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 05:05:54.533329 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.526773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.0.97.3:43945)
0927 05:05:54.526979 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.526984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.527066 (+ 82us) server_negotiation.cc:407] Connection header received
0927 05:05:54.527321 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.527327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.527443 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.527597 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:54.527958 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.529134 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.530477 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.531189 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.531346 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.531857 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:54.532661 (+ 804us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:54.532819 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.532935 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.533029 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 05:05:54.541873 2291 raft_consensus.cc:1261] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Refusing update from remote peer 19bb670db53f43f09b3b0ac82dc4dd5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:54.541873 2431 raft_consensus.cc:1261] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Refusing update from remote peer 19bb670db53f43f09b3b0ac82dc4dd5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:54.542943 2650 consensus_queue.cc:1035] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:54.543390 2648 consensus_queue.cc:1035] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:54.557900 2647 tablet_replica.cc:439] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: TxnStatusTablet state changed. Reason: New leader 19bb670db53f43f09b3b0ac82dc4dd5f. Latest consensus state: current_term: 1 leader_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } } }
I20260927 05:05:54.558367 2647 tablet_replica.cc:442] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:54.559808 2646 tablet_replica.cc:439] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: TxnStatusTablet state changed. Reason: New leader 19bb670db53f43f09b3b0ac82dc4dd5f. Latest consensus state: current_term: 1 leader_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } } }
I20260927 05:05:54.560166 2646 tablet_replica.cc:442] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:54.565231 2667 mvcc.cc:205] Tried to move back new op lower bound from 7333828831392317440 to 7333828831150366720. Current Snapshot: MvccSnapshot[applied={T|T < 7333828831392317440}]
I20260927 05:05:54.565253 2666 mvcc.cc:205] Tried to move back new op lower bound from 7333828831392317440 to 7333828831150366720. Current Snapshot: MvccSnapshot[applied={T|T < 7333828831392317440}]
I20260927 05:05:54.568117 2665 mvcc.cc:205] Tried to move back new op lower bound from 7333828831392317440 to 7333828831150366720. Current Snapshot: MvccSnapshot[applied={T|T < 7333828831392317440}]
I20260927 05:05:54.572853 2646 tablet_replica.cc:439] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } } }
I20260927 05:05:54.573128 2646 tablet_replica.cc:442] T db72f2bb8d0e414b8fec22aff03c95aa P dff604d7d023462d8224a33e443c85e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:54.577751 2648 tablet_replica.cc:439] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } } }
I20260927 05:05:54.578066 2648 tablet_replica.cc:442] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:05:54.582046 2648 tablet_replica.cc:439] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } } }
I20260927 05:05:54.582494 2648 tablet_replica.cc:442] T db72f2bb8d0e414b8fec22aff03c95aa P 19bb670db53f43f09b3b0ac82dc4dd5f: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:05:54.591929 2647 tablet_replica.cc:439] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } } }
I20260927 05:05:54.592291 2647 tablet_replica.cc:442] T db72f2bb8d0e414b8fec22aff03c95aa P 350b0feacf8d4533b286a1de137550ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:05:54.602125 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.596375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.97.3:43945)
0927 05:05:54.596578 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.596583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.596602 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:05:54.596806 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.596810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.596878 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.597028 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:54.597433 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.598268 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.599906 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.600601 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.600768 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.601260 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:54.601447 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:54.601564 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.601711 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.601797 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:05:54.621167 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.614958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33875 (local address 127.0.97.3:43945)
0927 05:05:54.615197 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.615205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.615463 (+ 258us) server_negotiation.cc:407] Connection header received
0927 05:05:54.615713 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.615722 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.615781 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.615883 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:54.616494 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.617367 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.619013 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.619524 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.619681 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.620269 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:54.620438 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:54.620552 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.620672 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.620803 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":84}
I20260927 05:05:54.621325 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.614612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43945 (local address 127.0.97.3:33875)
0927 05:05:54.615347 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.615365 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.615565 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.615894 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.615899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.615909 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:05:54.616319 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.616330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.617522 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.617527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.618837 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.618852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.619759 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.619766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.619901 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.620026 (+ 125us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:05:54.620551 (+ 525us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:05:54.620566 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.620652 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.620883 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":49,"thread_start_us":409,"threads_started":1}
I20260927 05:05:54.627025 2650 consensus_queue.cc:1035] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:54.634759 2291 raft_consensus.cc:1261] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Refusing update from remote peer 19bb670db53f43f09b3b0ac82dc4dd5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:54.635656 2670 consensus_queue.cc:1035] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:54.699542 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:54.699844 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:54.703717 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276: Sending DeleteTablet for 3 replicas of tablet 424b81e6fe9a4500aca20b9d39d0188a
I20260927 05:05:54.705096 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 424b81e6fe9a4500aca20b9d39d0188a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46436
I20260927 05:05:54.705132 2271 tablet_service.cc:1514] Processing DeleteTablet for tablet 424b81e6fe9a4500aca20b9d39d0188a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34560
I20260927 05:05:54.705158 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet 424b81e6fe9a4500aca20b9d39d0188a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53066
I20260927 05:05:54.707402 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:05:54.708567 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:05:54.717015 2683 tablet_replica.cc:330] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: stopping tablet replica
I20260927 05:05:54.717597 2682 tablet_replica.cc:330] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: stopping tablet replica
I20260927 05:05:54.717839 2683 raft_consensus.cc:2229] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:05:54.718009 2682 raft_consensus.cc:2229] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:54.718593 2270 tablet_service.cc:1467] Processing CreateTablet for tablet 26ba7b32a45445deb86c4ee29d4a78e7 (DEFAULT_TABLE table=test-table [id=4420d9ed5f924795ba2e69e5a50e7137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:54.719113 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ba7b32a45445deb86c4ee29d4a78e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:54.719051 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 26ba7b32a45445deb86c4ee29d4a78e7 (DEFAULT_TABLE table=test-table [id=4420d9ed5f924795ba2e69e5a50e7137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:54.719190 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 26ba7b32a45445deb86c4ee29d4a78e7 (DEFAULT_TABLE table=test-table [id=4420d9ed5f924795ba2e69e5a50e7137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:54.719610 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ba7b32a45445deb86c4ee29d4a78e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:54.719632 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ba7b32a45445deb86c4ee29d4a78e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:54.725782 2640 tablet_bootstrap.cc:492] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0: Bootstrap starting.
I20260927 05:05:54.726950 2683 raft_consensus.cc:2258] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:54.727553 2640 tablet_bootstrap.cc:654] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:54.728950 2641 tablet_bootstrap.cc:492] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad: Bootstrap starting.
I20260927 05:05:54.730594 2641 tablet_bootstrap.cc:654] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:54.733980 2681 tablet_replica.cc:330] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: stopping tablet replica
I20260927 05:05:54.735435 2681 raft_consensus.cc:2229] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:54.735828 2681 raft_consensus.cc:2258] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:54.736706 2642 tablet_bootstrap.cc:492] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f: Bootstrap starting.
I20260927 05:05:54.738921 2642 tablet_bootstrap.cc:654] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:54.739442 2681 ts_tablet_manager.cc:1905] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:54.746315 2683 ts_tablet_manager.cc:1905] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:54.748600 2682 raft_consensus.cc:2258] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:54.751834 2682 ts_tablet_manager.cc:1905] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:54.754122 2681 ts_tablet_manager.cc:1918] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:05:54.754331 2681 log.cc:1196] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/424b81e6fe9a4500aca20b9d39d0188a
I20260927 05:05:54.754987 2681 ts_tablet_manager.cc:1939] T 424b81e6fe9a4500aca20b9d39d0188a P dff604d7d023462d8224a33e443c85e0: Deleting consensus metadata
I20260927 05:05:54.757375 2118 catalog_manager.cc:4841] TS dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525): tablet 424b81e6fe9a4500aca20b9d39d0188a (table test-table [id=42bc03e79cf5472e9a9ae10d93f245f2]) successfully deleted
I20260927 05:05:54.760857 2641 tablet_bootstrap.cc:492] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad: No bootstrap required, opened a new log
I20260927 05:05:54.761041 2641 ts_tablet_manager.cc:1397] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.009s
I20260927 05:05:54.761837 2641 raft_consensus.cc:348] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.762097 2641 raft_consensus.cc:374] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:54.762179 2641 raft_consensus.cc:729] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350b0feacf8d4533b286a1de137550ad, State: Initialized, Role: FOLLOWER
I20260927 05:05:54.764802 2642 tablet_bootstrap.cc:492] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f: No bootstrap required, opened a new log
I20260927 05:05:54.764985 2642 ts_tablet_manager.cc:1397] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.014s
I20260927 05:05:54.765502 2641 consensus_queue.cc:260] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.765909 2642 raft_consensus.cc:348] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.766155 2642 raft_consensus.cc:374] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:54.766249 2642 raft_consensus.cc:729] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bb670db53f43f09b3b0ac82dc4dd5f, State: Initialized, Role: FOLLOWER
I20260927 05:05:54.766531 2642 consensus_queue.cc:260] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.766927 2641 ts_tablet_manager.cc:1428] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260927 05:05:54.769220 2640 tablet_bootstrap.cc:492] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0: No bootstrap required, opened a new log
I20260927 05:05:54.769518 2640 ts_tablet_manager.cc:1397] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.007s
I20260927 05:05:54.770274 2640 raft_consensus.cc:348] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.770541 2640 raft_consensus.cc:374] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:54.770625 2640 raft_consensus.cc:729] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff604d7d023462d8224a33e443c85e0, State: Initialized, Role: FOLLOWER
I20260927 05:05:54.770855 2640 consensus_queue.cc:260] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.771733 2642 ts_tablet_manager.cc:1428] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260927 05:05:54.773500 2640 ts_tablet_manager.cc:1428] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260927 05:05:54.775600 2682 ts_tablet_manager.cc:1918] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:05:54.775801 2682 log.cc:1196] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/424b81e6fe9a4500aca20b9d39d0188a
I20260927 05:05:54.776444 2682 ts_tablet_manager.cc:1939] T 424b81e6fe9a4500aca20b9d39d0188a P 350b0feacf8d4533b286a1de137550ad: Deleting consensus metadata
I20260927 05:05:54.777058 2683 ts_tablet_manager.cc:1918] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:05:54.777263 2683 log.cc:1196] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/424b81e6fe9a4500aca20b9d39d0188a
I20260927 05:05:54.777786 2683 ts_tablet_manager.cc:1939] T 424b81e6fe9a4500aca20b9d39d0188a P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting consensus metadata
I20260927 05:05:54.778120 2118 catalog_manager.cc:4841] TS 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411): tablet 424b81e6fe9a4500aca20b9d39d0188a (table test-table [id=42bc03e79cf5472e9a9ae10d93f245f2]) successfully deleted
I20260927 05:05:54.779290 2121 catalog_manager.cc:4841] TS 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945): tablet 424b81e6fe9a4500aca20b9d39d0188a (table test-table [id=42bc03e79cf5472e9a9ae10d93f245f2]) successfully deleted
I20260927 05:05:54.790524 2646 raft_consensus.cc:482] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:05:54.790776 2646 raft_consensus.cc:504] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.792925 2646 leader_election.cc:288] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411), 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945)
I20260927 05:05:54.811422 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.793142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43411 (local address 127.0.97.1:55305)
0927 05:05:54.804409 (+ 11267us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.804432 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.804632 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.805053 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.805059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.805088 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:54.805481 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.805492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.806852 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.806862 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.809001 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.809015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.810526 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.810533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.810737 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.810773 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.810877 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.811027 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11108,"mutex_wait_us":337,"thread_start_us":10275,"threads_started":1}
I20260927 05:05:54.811483 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.793384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55305 (local address 127.0.97.2:43411)
0927 05:05:54.804391 (+ 11007us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.804399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.804546 (+ 147us) server_negotiation.cc:407] Connection header received
0927 05:05:54.804801 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.804806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.804880 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.805007 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:54.805688 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.806682 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.809200 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.810342 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.810478 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.810585 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.810919 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.811077 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10854,"thread_start_us":10713,"threads_started":1}
I20260927 05:05:54.812431 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.793490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49989 (local address 127.0.97.3:43945)
0927 05:05:54.793749 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.793759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.803467 (+ 9708us) server_negotiation.cc:407] Connection header received
0927 05:05:54.803761 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.803768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.803861 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.804053 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:54.805105 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.806565 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.809715 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.811140 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.811282 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.811388 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.812000 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.812159 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 05:05:54.812489 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.793142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43945 (local address 127.0.97.1:49989)
0927 05:05:54.803272 (+ 10130us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.803316 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.803561 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.804352 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.804359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.804403 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:54.804839 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.804855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.806801 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.806807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.809530 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.809546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.811553 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.811575 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.811764 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.811801 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.811922 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.812101 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9951,"mutex_wait_us":128,"thread_start_us":9728,"threads_started":1}
I20260927 05:05:54.812778 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ba7b32a45445deb86c4ee29d4a78e7" candidate_uuid: "dff604d7d023462d8224a33e443c85e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350b0feacf8d4533b286a1de137550ad" is_pre_election: true
I20260927 05:05:54.813108 2430 raft_consensus.cc:2454] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff604d7d023462d8224a33e443c85e0 in term 0.
I20260927 05:05:54.813472 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ba7b32a45445deb86c4ee29d4a78e7" candidate_uuid: "dff604d7d023462d8224a33e443c85e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" is_pre_election: true
I20260927 05:05:54.813684 2221 leader_election.cc:302] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 350b0feacf8d4533b286a1de137550ad, dff604d7d023462d8224a33e443c85e0; no voters:
I20260927 05:05:54.813923 2571 raft_consensus.cc:2454] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff604d7d023462d8224a33e443c85e0 in term 0.
I20260927 05:05:54.814164 2646 raft_consensus.cc:2781] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:05:54.814289 2646 raft_consensus.cc:482] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:05:54.814397 2646 raft_consensus.cc:3037] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:54.817122 2646 raft_consensus.cc:504] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.817752 2646 leader_election.cc:288] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [CANDIDATE]: Term 1 election: Requested vote from peers 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411), 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945)
I20260927 05:05:54.818275 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ba7b32a45445deb86c4ee29d4a78e7" candidate_uuid: "dff604d7d023462d8224a33e443c85e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350b0feacf8d4533b286a1de137550ad"
I20260927 05:05:54.818382 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ba7b32a45445deb86c4ee29d4a78e7" candidate_uuid: "dff604d7d023462d8224a33e443c85e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f"
I20260927 05:05:54.818519 2430 raft_consensus.cc:3037] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:54.818589 2571 raft_consensus.cc:3037] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:54.821973 2430 raft_consensus.cc:2454] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff604d7d023462d8224a33e443c85e0 in term 1.
I20260927 05:05:54.822329 2571 raft_consensus.cc:2454] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff604d7d023462d8224a33e443c85e0 in term 1.
I20260927 05:05:54.822607 2221 leader_election.cc:302] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 350b0feacf8d4533b286a1de137550ad, dff604d7d023462d8224a33e443c85e0; no voters:
I20260927 05:05:54.823143 2646 raft_consensus.cc:2781] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:54.823805 2646 raft_consensus.cc:686] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 1 LEADER]: Becoming Leader. State: Replica: dff604d7d023462d8224a33e443c85e0, State: Running, Role: LEADER
I20260927 05:05:54.824333 2646 consensus_queue.cc:237] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:54.828883 2133 catalog_manager.cc:5495] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff604d7d023462d8224a33e443c85e0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "dff604d7d023462d8224a33e443c85e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } health_report { overall_health: UNKNOWN } } }
I20260927 05:05:54.862358 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.851875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.0.97.1:41525)
0927 05:05:54.856329 (+ 4454us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.856334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.856357 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:05:54.856446 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.856450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.856529 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.856696 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:54.857213 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.857962 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.859996 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.860385 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.860484 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.861062 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:54.861670 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:54.861778 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.861928 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.861995 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4341,"thread_start_us":4243,"threads_started":1}
I20260927 05:05:54.879362 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.873405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43945 (local address 127.0.97.1:47287)
0927 05:05:54.874031 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.874047 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.874224 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.874579 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.874583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.874595 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:05:54.874876 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.874885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.876001 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.876006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.877189 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.877201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.877980 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.877986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.878130 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.878191 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:05:54.878808 (+ 617us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:05:54.878827 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.878905 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.879020 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":329,"threads_started":1}
I20260927 05:05:54.879382 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.873512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47287 (local address 127.0.97.3:43945)
0927 05:05:54.873719 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.873727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.874141 (+ 414us) server_negotiation.cc:407] Connection header received
0927 05:05:54.874370 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.874374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.874439 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.874567 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:54.875029 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.875815 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.877339 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.877790 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.877941 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.878451 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:54.878603 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:54.878744 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.878960 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.879054 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:05:54.889542 2571 raft_consensus.cc:1261] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Refusing update from remote peer dff604d7d023462d8224a33e443c85e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:54.890246 2430 raft_consensus.cc:1261] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Refusing update from remote peer dff604d7d023462d8224a33e443c85e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:54.890443 2690 consensus_queue.cc:1035] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:54.890877 2646 consensus_queue.cc:1035] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:54.932089 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.923257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41525 (local address 127.0.97.3:42743)
0927 05:05:54.923671 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:54.923700 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:54.923828 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:54.924952 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:54.924960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:54.924980 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:05:54.925260 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.925275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.927173 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.927182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.929380 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:54.929403 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.930415 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.930425 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:54.930640 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.930702 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:05:54.931209 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:05:54.931224 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 05:05:54.931325 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 05:05:54.931545 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260927 05:05:54.932857 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:54.924235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42743 (local address 127.0.97.1:41525)
0927 05:05:54.924399 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:54.924405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:54.924423 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:05:54.924489 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:54.924493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:54.924619 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:54.924736 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:54.925426 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.927014 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.929617 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:54.930184 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:54.930310 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:54.930888 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:54.931048 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:54.931149 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:54.932341 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0927 05:05:54.932431 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 05:05:54.934105 2666 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 05:05:56.533260 2134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:56.533457 2134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:56.537285 2134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276: Sending DeleteTablet for 3 replicas of tablet 26ba7b32a45445deb86c4ee29d4a78e7
I20260927 05:05:56.538092 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ba7b32a45445deb86c4ee29d4a78e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34560
I20260927 05:05:56.538164 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ba7b32a45445deb86c4ee29d4a78e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53066
I20260927 05:05:56.538309 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ba7b32a45445deb86c4ee29d4a78e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46436
I20260927 05:05:56.547133 2723 tablet_replica.cc:330] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0: stopping tablet replica
I20260927 05:05:56.547405 2723 raft_consensus.cc:2229] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:05:56.548664 2723 raft_consensus.cc:2258] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:56.549997 2723 ts_tablet_manager.cc:1905] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:56.554376 2725 tablet_replica.cc:330] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f: stopping tablet replica
I20260927 05:05:56.554665 2725 raft_consensus.cc:2229] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:56.554963 2725 raft_consensus.cc:2258] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:56.556156 2725 ts_tablet_manager.cc:1905] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:56.559409 2723 ts_tablet_manager.cc:1918] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:05:56.559571 2723 log.cc:1196] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/26ba7b32a45445deb86c4ee29d4a78e7
I20260927 05:05:56.560057 2724 tablet_replica.cc:330] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad: stopping tablet replica
I20260927 05:05:56.560094 2723 ts_tablet_manager.cc:1939] T 26ba7b32a45445deb86c4ee29d4a78e7 P dff604d7d023462d8224a33e443c85e0: Deleting consensus metadata
I20260927 05:05:56.560339 2724 raft_consensus.cc:2229] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:56.560616 2724 raft_consensus.cc:2258] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:56.561352 2118 catalog_manager.cc:4841] TS dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525): tablet 26ba7b32a45445deb86c4ee29d4a78e7 (table test-table [id=4420d9ed5f924795ba2e69e5a50e7137]) successfully deleted
I20260927 05:05:56.561717 2724 ts_tablet_manager.cc:1905] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:56.563525 2725 ts_tablet_manager.cc:1918] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:05:56.563661 2725 log.cc:1196] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/26ba7b32a45445deb86c4ee29d4a78e7
I20260927 05:05:56.564177 2725 ts_tablet_manager.cc:1939] T 26ba7b32a45445deb86c4ee29d4a78e7 P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting consensus metadata
I20260927 05:05:56.565349 2121 catalog_manager.cc:4841] TS 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945): tablet 26ba7b32a45445deb86c4ee29d4a78e7 (table test-table [id=4420d9ed5f924795ba2e69e5a50e7137]) successfully deleted
I20260927 05:05:56.567385 2724 ts_tablet_manager.cc:1918] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:05:56.567552 2724 log.cc:1196] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/26ba7b32a45445deb86c4ee29d4a78e7
I20260927 05:05:56.568056 2724 ts_tablet_manager.cc:1939] T 26ba7b32a45445deb86c4ee29d4a78e7 P 350b0feacf8d4533b286a1de137550ad: Deleting consensus metadata
I20260927 05:05:56.569217 2118 catalog_manager.cc:4841] TS 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411): tablet 26ba7b32a45445deb86c4ee29d4a78e7 (table test-table [id=4420d9ed5f924795ba2e69e5a50e7137]) successfully deleted
Sep 27 05:05:56 dist-test-slave-hgs0 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485556, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 05:05:56 dist-test-slave-hgs0 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485556, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:05:56.598217 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:56.583890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.0.97.62:37061)
0927 05:05:56.584443 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:56.584449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:56.584551 (+ 102us) server_negotiation.cc:407] Connection header received
0927 05:05:56.584786 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:56.584791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:56.584868 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:56.584974 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:56.586285 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:56.587062 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:56.588475 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:56.588925 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:56.589060 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:56.593429 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:56.593434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:56.593437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:56.593468 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:56.596118 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:56.596724 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:56.596728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:56.596731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:56.596868 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:56.597168 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:56.597171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:56.597173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:56.597410 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:56.597577 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:56.597816 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 05:05:56.597963 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":251,"threads_started":1}
I20260927 05:05:56.601732 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:05:56.602324 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:05:56.608644 2270 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1c5a4152334bef897f6fa3b69950c6 (DEFAULT_TABLE table=test-table [id=f3e609c5393845b89b2992f9a866a4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:56.608711 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1c5a4152334bef897f6fa3b69950c6 (DEFAULT_TABLE table=test-table [id=f3e609c5393845b89b2992f9a866a4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:56.608881 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1c5a4152334bef897f6fa3b69950c6 (DEFAULT_TABLE table=test-table [id=f3e609c5393845b89b2992f9a866a4ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:56.609102 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1c5a4152334bef897f6fa3b69950c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:56.609143 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1c5a4152334bef897f6fa3b69950c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:56.609249 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1c5a4152334bef897f6fa3b69950c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:56.614755 2734 tablet_bootstrap.cc:492] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad: Bootstrap starting.
I20260927 05:05:56.616226 2736 tablet_bootstrap.cc:492] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f: Bootstrap starting.
I20260927 05:05:56.616628 2734 tablet_bootstrap.cc:654] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:56.618623 2736 tablet_bootstrap.cc:654] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:56.618822 2734 tablet_bootstrap.cc:492] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad: No bootstrap required, opened a new log
I20260927 05:05:56.618963 2734 ts_tablet_manager.cc:1397] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 05:05:56.618978 2735 tablet_bootstrap.cc:492] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0: Bootstrap starting.
I20260927 05:05:56.619828 2734 raft_consensus.cc:348] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.620209 2734 raft_consensus.cc:374] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:56.620296 2734 raft_consensus.cc:729] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350b0feacf8d4533b286a1de137550ad, State: Initialized, Role: FOLLOWER
I20260927 05:05:56.620594 2734 consensus_queue.cc:260] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.621145 2735 tablet_bootstrap.cc:654] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:56.621292 2736 tablet_bootstrap.cc:492] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f: No bootstrap required, opened a new log
I20260927 05:05:56.621425 2736 ts_tablet_manager.cc:1397] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 05:05:56.621460 2734 ts_tablet_manager.cc:1428] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:05:56.622128 2736 raft_consensus.cc:348] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.622375 2736 raft_consensus.cc:374] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:56.622459 2736 raft_consensus.cc:729] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bb670db53f43f09b3b0ac82dc4dd5f, State: Initialized, Role: FOLLOWER
I20260927 05:05:56.622727 2736 consensus_queue.cc:260] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.623270 2736 ts_tablet_manager.cc:1428] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:05:56.623502 2735 tablet_bootstrap.cc:492] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0: No bootstrap required, opened a new log
I20260927 05:05:56.623649 2735 ts_tablet_manager.cc:1397] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 05:05:56.624434 2735 raft_consensus.cc:348] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.624682 2735 raft_consensus.cc:374] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:56.624779 2735 raft_consensus.cc:729] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff604d7d023462d8224a33e443c85e0, State: Initialized, Role: FOLLOWER
I20260927 05:05:56.625005 2735 consensus_queue.cc:260] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.625638 2735 ts_tablet_manager.cc:1428] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 05:05:56.682156 2717 raft_consensus.cc:482] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:05:56.682374 2717 raft_consensus.cc:504] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.683110 2717 leader_election.cc:288] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411), 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945)
I20260927 05:05:56.683672 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1c5a4152334bef897f6fa3b69950c6" candidate_uuid: "dff604d7d023462d8224a33e443c85e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350b0feacf8d4533b286a1de137550ad" is_pre_election: true
I20260927 05:05:56.683671 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1c5a4152334bef897f6fa3b69950c6" candidate_uuid: "dff604d7d023462d8224a33e443c85e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" is_pre_election: true
I20260927 05:05:56.683908 2430 raft_consensus.cc:2454] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff604d7d023462d8224a33e443c85e0 in term 0.
I20260927 05:05:56.683908 2571 raft_consensus.cc:2454] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff604d7d023462d8224a33e443c85e0 in term 0.
I20260927 05:05:56.684453 2221 leader_election.cc:302] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19bb670db53f43f09b3b0ac82dc4dd5f, 350b0feacf8d4533b286a1de137550ad, dff604d7d023462d8224a33e443c85e0; no voters:
I20260927 05:05:56.684728 2717 raft_consensus.cc:2781] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:05:56.684814 2717 raft_consensus.cc:482] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:05:56.684909 2717 raft_consensus.cc:3037] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:56.687242 2717 raft_consensus.cc:504] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.687763 2717 leader_election.cc:288] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [CANDIDATE]: Term 1 election: Requested vote from peers 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411), 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945)
I20260927 05:05:56.688220 2430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1c5a4152334bef897f6fa3b69950c6" candidate_uuid: "dff604d7d023462d8224a33e443c85e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350b0feacf8d4533b286a1de137550ad"
I20260927 05:05:56.688273 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1c5a4152334bef897f6fa3b69950c6" candidate_uuid: "dff604d7d023462d8224a33e443c85e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f"
I20260927 05:05:56.688418 2430 raft_consensus.cc:3037] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:56.688457 2571 raft_consensus.cc:3037] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:56.691378 2430 raft_consensus.cc:2454] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff604d7d023462d8224a33e443c85e0 in term 1.
I20260927 05:05:56.691378 2571 raft_consensus.cc:2454] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff604d7d023462d8224a33e443c85e0 in term 1.
I20260927 05:05:56.691768 2221 leader_election.cc:302] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 350b0feacf8d4533b286a1de137550ad, dff604d7d023462d8224a33e443c85e0; no voters:
I20260927 05:05:56.692123 2717 raft_consensus.cc:2781] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:56.692298 2717 raft_consensus.cc:686] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 1 LEADER]: Becoming Leader. State: Replica: dff604d7d023462d8224a33e443c85e0, State: Running, Role: LEADER
I20260927 05:05:56.692564 2717 consensus_queue.cc:237] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.695188 2134 catalog_manager.cc:5495] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff604d7d023462d8224a33e443c85e0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "dff604d7d023462d8224a33e443c85e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } health_report { overall_health: UNKNOWN } } }
I20260927 05:05:56.725792 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:56.720963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43836 (local address 127.0.97.1:41525)
0927 05:05:56.721495 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:56.721499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:56.721517 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:05:56.721563 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:56.721567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:56.721643 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:56.721723 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:56.722022 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:56.722796 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:56.724083 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:56.724481 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:56.724565 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:56.724938 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:56.725031 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:56.725098 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:56.725431 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 05:05:56.725501 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":320,"threads_started":1}
I20260927 05:05:56.734534 2571 raft_consensus.cc:1261] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Refusing update from remote peer dff604d7d023462d8224a33e443c85e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:56.734678 2430 raft_consensus.cc:1261] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Refusing update from remote peer dff604d7d023462d8224a33e443c85e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:56.735185 2717 consensus_queue.cc:1035] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:56.735498 2700 consensus_queue.cc:1035] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:56.778584 2134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:56.778823 2134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:56.781726 2134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276: Sending DeleteTablet for 3 replicas of tablet 0f1c5a4152334bef897f6fa3b69950c6
I20260927 05:05:56.782379 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1c5a4152334bef897f6fa3b69950c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34560
I20260927 05:05:56.782481 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1c5a4152334bef897f6fa3b69950c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53066
I20260927 05:05:56.782673 2723 tablet_replica.cc:330] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0: stopping tablet replica
I20260927 05:05:56.782759 2724 tablet_replica.cc:330] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad: stopping tablet replica
I20260927 05:05:56.782866 2723 raft_consensus.cc:2229] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:05:56.782943 2724 raft_consensus.cc:2229] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:56.783129 2724 raft_consensus.cc:2258] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:56.783185 2723 raft_consensus.cc:2258] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:56.783193 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1c5a4152334bef897f6fa3b69950c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46436
I20260927 05:05:56.783435 2725 tablet_replica.cc:330] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f: stopping tablet replica
I20260927 05:05:56.783629 2725 raft_consensus.cc:2229] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:56.783502 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 05:05:56.783797 2725 raft_consensus.cc:2258] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 05:05:56.784192 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:05:56.784286 2724 ts_tablet_manager.cc:1905] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:56.784974 2725 ts_tablet_manager.cc:1905] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:56.790181 2724 ts_tablet_manager.cc:1918] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:05:56.790343 2724 log.cc:1196] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/0f1c5a4152334bef897f6fa3b69950c6
I20260927 05:05:56.790787 2724 ts_tablet_manager.cc:1939] T 0f1c5a4152334bef897f6fa3b69950c6 P 350b0feacf8d4533b286a1de137550ad: Deleting consensus metadata
I20260927 05:05:56.790999 2725 ts_tablet_manager.cc:1918] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:05:56.791188 2725 log.cc:1196] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/0f1c5a4152334bef897f6fa3b69950c6
I20260927 05:05:56.791211 2270 tablet_service.cc:1467] Processing CreateTablet for tablet a772f9583c4740ca88a50fac1a7920a2 (DEFAULT_TABLE table=test-table [id=2d1825d3d53c4837adad51f710987fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:56.791200 2551 tablet_service.cc:1467] Processing CreateTablet for tablet a772f9583c4740ca88a50fac1a7920a2 (DEFAULT_TABLE table=test-table [id=2d1825d3d53c4837adad51f710987fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:56.791657 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a772f9583c4740ca88a50fac1a7920a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:56.791651 2725 ts_tablet_manager.cc:1939] T 0f1c5a4152334bef897f6fa3b69950c6 P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting consensus metadata
I20260927 05:05:56.791694 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a772f9583c4740ca88a50fac1a7920a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:56.791819 2411 tablet_service.cc:1467] Processing CreateTablet for tablet a772f9583c4740ca88a50fac1a7920a2 (DEFAULT_TABLE table=test-table [id=2d1825d3d53c4837adad51f710987fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:05:56.792213 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a772f9583c4740ca88a50fac1a7920a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:05:56.793416 2723 ts_tablet_manager.cc:1905] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:56.797834 2736 tablet_bootstrap.cc:492] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f: Bootstrap starting.
I20260927 05:05:56.798785 2735 tablet_bootstrap.cc:492] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0: Bootstrap starting.
I20260927 05:05:56.799412 2736 tablet_bootstrap.cc:654] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:56.800148 2723 ts_tablet_manager.cc:1918] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:05:56.800315 2723 log.cc:1196] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/0f1c5a4152334bef897f6fa3b69950c6
I20260927 05:05:56.800614 2735 tablet_bootstrap.cc:654] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:56.800771 2723 ts_tablet_manager.cc:1939] T 0f1c5a4152334bef897f6fa3b69950c6 P dff604d7d023462d8224a33e443c85e0: Deleting consensus metadata
I20260927 05:05:56.802564 2121 catalog_manager.cc:4841] TS 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945): tablet 0f1c5a4152334bef897f6fa3b69950c6 (table test-table [id=f3e609c5393845b89b2992f9a866a4ca]) successfully deleted
I20260927 05:05:56.803619 2734 tablet_bootstrap.cc:492] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad: Bootstrap starting.
I20260927 05:05:56.804153 2118 catalog_manager.cc:4841] TS 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411): tablet 0f1c5a4152334bef897f6fa3b69950c6 (table test-table [id=f3e609c5393845b89b2992f9a866a4ca]) successfully deleted
I20260927 05:05:56.804564 2118 catalog_manager.cc:4841] TS dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525): tablet 0f1c5a4152334bef897f6fa3b69950c6 (table test-table [id=f3e609c5393845b89b2992f9a866a4ca]) successfully deleted
I20260927 05:05:56.805138 2734 tablet_bootstrap.cc:654] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad: Neither blocks nor log segments found. Creating new log.
I20260927 05:05:56.805313 2735 tablet_bootstrap.cc:492] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0: No bootstrap required, opened a new log
I20260927 05:05:56.805495 2735 ts_tablet_manager.cc:1397] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 05:05:56.806404 2735 raft_consensus.cc:348] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.806740 2735 raft_consensus.cc:374] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:56.806838 2735 raft_consensus.cc:729] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff604d7d023462d8224a33e443c85e0, State: Initialized, Role: FOLLOWER
I20260927 05:05:56.807044 2734 tablet_bootstrap.cc:492] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad: No bootstrap required, opened a new log
I20260927 05:05:56.807191 2734 ts_tablet_manager.cc:1397] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 05:05:56.807120 2735 consensus_queue.cc:260] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.807832 2735 ts_tablet_manager.cc:1428] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 05:05:56.807850 2734 raft_consensus.cc:348] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.808162 2734 raft_consensus.cc:374] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:56.808243 2734 raft_consensus.cc:729] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350b0feacf8d4533b286a1de137550ad, State: Initialized, Role: FOLLOWER
I20260927 05:05:56.808444 2734 consensus_queue.cc:260] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.809016 2734 ts_tablet_manager.cc:1428] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:05:56.812441 2736 tablet_bootstrap.cc:492] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f: No bootstrap required, opened a new log
I20260927 05:05:56.812677 2736 ts_tablet_manager.cc:1397] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260927 05:05:56.813563 2736 raft_consensus.cc:348] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.813855 2736 raft_consensus.cc:374] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:05:56.813949 2736 raft_consensus.cc:729] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bb670db53f43f09b3b0ac82dc4dd5f, State: Initialized, Role: FOLLOWER
I20260927 05:05:56.814229 2736 consensus_queue.cc:260] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:56.814757 2736 ts_tablet_manager.cc:1428] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 05:05:57.018361 2739 raft_consensus.cc:482] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:05:57.018592 2739 raft_consensus.cc:504] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:57.020066 2739 leader_election.cc:288] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525), 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945)
I20260927 05:05:57.029084 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:57.020433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39485 (local address 127.0.97.3:43945)
0927 05:05:57.021288 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:57.021294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:57.021546 (+ 252us) server_negotiation.cc:407] Connection header received
0927 05:05:57.021678 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:57.021682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:57.021758 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:57.021878 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:57.023054 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.024162 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.026701 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.027962 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.028091 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:57.028215 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:57.028677 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 05:05:57.028801 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":508,"threads_started":1}
I20260927 05:05:57.029054 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:57.020476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43863 (local address 127.0.97.1:41525)
0927 05:05:57.020693 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:57.020699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:57.021097 (+ 398us) server_negotiation.cc:407] Connection header received
0927 05:05:57.021445 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:57.021448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:57.021535 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:57.021637 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:05:57.022679 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.023876 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.026666 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.027963 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.028101 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:57.028193 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:57.028678 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 05:05:57.028801 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:05:57.029294 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:57.020303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41525 (local address 127.0.97.2:43863)
0927 05:05:57.020990 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:57.021009 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:57.021265 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:57.021664 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:57.021669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:57.021690 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:57.022471 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:57.022486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.024055 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.024059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:57.026500 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:57.026512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.028169 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.028174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:57.028448 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:57.028526 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 05:05:57.028632 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 05:05:57.028798 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":203,"thread_start_us":296,"threads_started":1}
I20260927 05:05:57.029315 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:57.020303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43945 (local address 127.0.97.2:39485)
0927 05:05:57.021461 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:57.021474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:57.021559 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:57.022337 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:57.022343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:57.022370 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:05:57.022870 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:57.022883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.024270 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.024275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:57.026499 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:57.026520 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.028146 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.028158 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:57.028447 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:57.028531 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 05:05:57.028632 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 05:05:57.028798 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":569,"thread_start_us":488,"threads_started":1}
I20260927 05:05:57.029902 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a772f9583c4740ca88a50fac1a7920a2" candidate_uuid: "350b0feacf8d4533b286a1de137550ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff604d7d023462d8224a33e443c85e0" is_pre_election: true
I20260927 05:05:57.030035 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a772f9583c4740ca88a50fac1a7920a2" candidate_uuid: "350b0feacf8d4533b286a1de137550ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" is_pre_election: true
I20260927 05:05:57.030148 2291 raft_consensus.cc:2454] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 350b0feacf8d4533b286a1de137550ad in term 0.
I20260927 05:05:57.030352 2571 raft_consensus.cc:2454] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 350b0feacf8d4533b286a1de137550ad in term 0.
I20260927 05:05:57.030676 2361 leader_election.cc:302] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 350b0feacf8d4533b286a1de137550ad, dff604d7d023462d8224a33e443c85e0; no voters:
I20260927 05:05:57.031057 2739 raft_consensus.cc:2781] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:05:57.031188 2739 raft_consensus.cc:482] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:05:57.031277 2739 raft_consensus.cc:3037] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:57.033398 2739 raft_consensus.cc:504] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:57.033947 2739 leader_election.cc:288] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [CANDIDATE]: Term 1 election: Requested vote from peers dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525), 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945)
I20260927 05:05:57.034394 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a772f9583c4740ca88a50fac1a7920a2" candidate_uuid: "350b0feacf8d4533b286a1de137550ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f"
I20260927 05:05:57.034431 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a772f9583c4740ca88a50fac1a7920a2" candidate_uuid: "350b0feacf8d4533b286a1de137550ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff604d7d023462d8224a33e443c85e0"
I20260927 05:05:57.034562 2571 raft_consensus.cc:3037] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:57.034687 2291 raft_consensus.cc:3037] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:05:57.036998 2571 raft_consensus.cc:2454] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 350b0feacf8d4533b286a1de137550ad in term 1.
I20260927 05:05:57.037359 2291 raft_consensus.cc:2454] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 350b0feacf8d4533b286a1de137550ad in term 1.
I20260927 05:05:57.037393 2364 leader_election.cc:302] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bb670db53f43f09b3b0ac82dc4dd5f, 350b0feacf8d4533b286a1de137550ad; no voters:
I20260927 05:05:57.037637 2739 raft_consensus.cc:2781] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:05:57.038064 2739 raft_consensus.cc:686] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 1 LEADER]: Becoming Leader. State: Replica: 350b0feacf8d4533b286a1de137550ad, State: Running, Role: LEADER
I20260927 05:05:57.038417 2739 consensus_queue.cc:237] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } }
I20260927 05:05:57.041951 2132 catalog_manager.cc:5495] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 350b0feacf8d4533b286a1de137550ad (127.0.97.2). New cstate: current_term: 1 leader_uuid: "350b0feacf8d4533b286a1de137550ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "350b0feacf8d4533b286a1de137550ad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 } health_report { overall_health: UNKNOWN } } }
I20260927 05:05:57.065258 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:57.058086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.0.97.2:43411)
0927 05:05:57.058712 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:57.058719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:57.058744 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:05:57.058850 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:57.058856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:57.058959 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:57.059153 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:57.059964 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.061200 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.062469 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.063175 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.063307 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:57.063682 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:57.064525 (+ 843us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:57.064680 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:57.064744 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 05:05:57.064841 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":348,"threads_started":1}
I20260927 05:05:57.080968 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:57.075341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59759 (local address 127.0.97.3:43945)
0927 05:05:57.075510 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:57.075515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:57.075865 (+ 350us) server_negotiation.cc:407] Connection header received
0927 05:05:57.076081 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:57.076084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:57.076145 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:57.076217 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:57.076745 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.077399 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.079291 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.079820 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.079909 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:57.080373 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:57.080466 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:57.080532 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:57.080675 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 05:05:57.080732 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:05:57.081252 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:57.075076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43945 (local address 127.0.97.2:59759)
0927 05:05:57.075775 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:57.075792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:57.075910 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:57.076265 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:57.076268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:57.076278 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:05:57.076605 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:57.076614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.077564 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.077568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:57.079143 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:57.079156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.080027 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.080033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:57.080151 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:57.080203 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:05:57.080570 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:05:57.080578 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 05:05:57.080647 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 05:05:57.080748 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":346,"threads_started":1}
I20260927 05:05:57.088953 2291 raft_consensus.cc:1261] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Refusing update from remote peer 350b0feacf8d4533b286a1de137550ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:57.089092 2571 raft_consensus.cc:1261] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Refusing update from remote peer 350b0feacf8d4533b286a1de137550ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:05:57.089689 2753 consensus_queue.cc:1035] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff604d7d023462d8224a33e443c85e0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:57.090051 2739 consensus_queue.cc:1035] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "19bb670db53f43f09b3b0ac82dc4dd5f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:05:57.117060 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:57.111743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41325 (local address 127.0.97.2:43411)
0927 05:05:57.111943 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:57.111971 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:57.112335 (+ 364us) server_negotiation.cc:407] Connection header received
0927 05:05:57.112541 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:57.112546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:57.112618 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:57.112710 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:05:57.113125 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.113906 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.115278 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.115747 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.115906 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:57.116356 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:05:57.116461 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:05:57.116577 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:57.116684 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 05:05:57.116761 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:05:57.117172 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:57.111697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43411 (local address 127.0.97.3:41325)
0927 05:05:57.112239 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 05:05:57.112255 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:05:57.112403 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:05:57.112721 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:05:57.112726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:05:57.112736 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:05:57.112982 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:57.112990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.114051 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.114057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:57.115133 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:05:57.115144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:05:57.115927 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:05:57.115955 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:05:57.116159 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:57.116195 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:05:57.116579 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:05:57.116594 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 05:05:57.116667 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 05:05:57.116814 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":307,"threads_started":1}
W20260927 05:05:57.118165 2704 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 05:05:58.708786 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:58.708994 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:05:58.712620 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76469c560eb64222925acbc9a0052276: Sending DeleteTablet for 3 replicas of tablet a772f9583c4740ca88a50fac1a7920a2
I20260927 05:05:58.713341 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet a772f9583c4740ca88a50fac1a7920a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53066
I20260927 05:05:58.713382 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet a772f9583c4740ca88a50fac1a7920a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34560
I20260927 05:05:58.713368 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet a772f9583c4740ca88a50fac1a7920a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:05:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46436
I20260927 05:05:58.714012 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 2205
I20260927 05:05:58.725797 2790 tablet_replica.cc:330] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad: stopping tablet replica
I20260927 05:05:58.726166 2790 raft_consensus.cc:2229] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:05:58.726855 2790 raft_consensus.cc:2258] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:58.728771 2789 tablet_replica.cc:330] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0: stopping tablet replica
I20260927 05:05:58.728921 2791 tablet_replica.cc:330] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f: stopping tablet replica
I20260927 05:05:58.728902 2790 ts_tablet_manager.cc:1905] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:58.729084 2789 raft_consensus.cc:2229] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:58.729172 2791 raft_consensus.cc:2229] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:05:58.729391 2789 raft_consensus.cc:2258] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:58.729434 2791 raft_consensus.cc:2258] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:05:58.730716 2789 ts_tablet_manager.cc:1905] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:58.730799 2791 ts_tablet_manager.cc:1905] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:05:58.735124 2790 ts_tablet_manager.cc:1918] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:05:58.735314 2790 log.cc:1196] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/a772f9583c4740ca88a50fac1a7920a2
I20260927 05:05:58.735828 2790 ts_tablet_manager.cc:1939] T a772f9583c4740ca88a50fac1a7920a2 P 350b0feacf8d4533b286a1de137550ad: Deleting consensus metadata
I20260927 05:05:58.737000 2118 catalog_manager.cc:4841] TS 350b0feacf8d4533b286a1de137550ad (127.0.97.2:43411): tablet a772f9583c4740ca88a50fac1a7920a2 (table test-table [id=2d1825d3d53c4837adad51f710987fac]) successfully deleted
I20260927 05:05:58.737341 2789 ts_tablet_manager.cc:1918] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:05:58.737473 2789 log.cc:1196] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/a772f9583c4740ca88a50fac1a7920a2
I20260927 05:05:58.737970 2789 ts_tablet_manager.cc:1939] T a772f9583c4740ca88a50fac1a7920a2 P dff604d7d023462d8224a33e443c85e0: Deleting consensus metadata
I20260927 05:05:58.738339 2791 ts_tablet_manager.cc:1918] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 05:05:58.738484 2791 log.cc:1196] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/a772f9583c4740ca88a50fac1a7920a2
I20260927 05:05:58.738910 2791 ts_tablet_manager.cc:1939] T a772f9583c4740ca88a50fac1a7920a2 P 19bb670db53f43f09b3b0ac82dc4dd5f: Deleting consensus metadata
I20260927 05:05:58.739112 2118 catalog_manager.cc:4841] TS dff604d7d023462d8224a33e443c85e0 (127.0.97.1:41525): tablet a772f9583c4740ca88a50fac1a7920a2 (table test-table [id=2d1825d3d53c4837adad51f710987fac]) successfully deleted
I20260927 05:05:58.739852 2121 catalog_manager.cc:4841] TS 19bb670db53f43f09b3b0ac82dc4dd5f (127.0.97.3:43945): tablet a772f9583c4740ca88a50fac1a7920a2 (table test-table [id=2d1825d3d53c4837adad51f710987fac]) successfully deleted
Sep 27 05:05:58 dist-test-slave-hgs0 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485556, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:05:58.746915 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:58.716068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.0.97.1:41525)
0927 05:05:58.727561 (+ 11493us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:58.727571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:58.730121 (+ 2550us) server_negotiation.cc:407] Connection header received
0927 05:05:58.730301 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:58.730306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:58.730401 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:58.730518 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:58.732489 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:58.733698 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:58.735485 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:58.736195 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:58.736318 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:58.741274 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:58.741292 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:58.741313 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:58.741354 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:58.744666 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:58.745232 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:58.745247 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:58.745250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:58.745374 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:58.745661 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:58.745664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:58.745666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:58.745966 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:58.746168 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:58.746380 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 05:05:58.746526 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11321,"thread_start_us":11193,"threads_started":1}
W20260927 05:05:58.747505 2332 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43846
W20260927 05:05:58.748158 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 2205: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:05:58.748390 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 2205
I20260927 05:05:58.771015 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 2345
Sep 27 05:05:58 dist-test-slave-hgs0 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485556, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:05:58.785663 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:58.771762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.0.97.2:43411)
0927 05:05:58.772288 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:58.772292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:58.772306 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:05:58.772356 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:58.772359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:58.772410 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:58.772558 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:58.773899 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:58.774574 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:58.775772 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:58.776345 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:58.776472 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:58.780599 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:58.780613 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:58.780629 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:58.780659 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:58.783242 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:58.783804 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:58.783810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:58.783813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:58.783931 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:58.784328 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:58.784331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:58.784333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:58.784666 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:58.784867 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:58.785145 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 05:05:58.785326 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":312,"threads_started":1}
W20260927 05:05:58.786273 2476 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55702
W20260927 05:05:58.786881 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 2345: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:05:58.787031 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 2345
I20260927 05:05:58.808481 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 2485
Sep 27 05:05:58 dist-test-slave-hgs0 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485556, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:05:58.822532 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:05:58.809117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.0.97.3:43945)
0927 05:05:58.809610 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 05:05:58.809615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:05:58.809629 (+ 14us) server_negotiation.cc:407] Connection header received
0927 05:05:58.809679 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:05:58.809682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:05:58.809741 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:05:58.809830 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:05:58.811016 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:58.811733 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:58.812987 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:05:58.813376 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:05:58.813465 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:05:58.817705 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:05:58.817718 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:05:58.817734 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:05:58.817766 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:05:58.820406 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:58.820977 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:58.820983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:58.820986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:58.821113 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:05:58.821407 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:05:58.821410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:05:58.821411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:05:58.821720 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:05:58.821927 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 05:05:58.822141 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 05:05:58.822286 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":294,"threads_started":1}
W20260927 05:05:58.823204 2616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34852
W20260927 05:05:58.823794 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 2485: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:05:58.824097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 2485
I20260927 05:05:58.848649 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 2102
I20260927 05:05:58.849758 2174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:05:58.963518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 2102
2026-09-27T05:05:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8524 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:05:59 dist-test-slave-hgs0 krb5kdc[2799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:05:59 dist-test-slave-hgs0 krb5kdc[2799](info): set up 2 sockets
Sep 27 05:05:59 dist-test-slave-hgs0 krb5kdc[2799](info): commencing operation
krb5kdc: starting...
W20260927 05:06:00.052383 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.008s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:06:00 dist-test-slave-hgs0 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:06:00.240916 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35893
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:00.550971 2812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:00.551275 2812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:00.551366 2812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:00.558506 2812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:00.558621 2812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:00.558668 2812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:00.558753 2812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:00.558801 2812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:00.568724 2812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:00.569980 2812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:00.571241 2812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:00.583433 2817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:00.583375 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:00.583408 2818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:00.584038 2812 server_base.cc:1034] running on GCE node
I20260927 05:06:00.584942 2812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:00.586412 2812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:00.587612 2812 hybrid_clock.cc:648] HybridClock initialized: now 1790485560587545 us; error 56 us; skew 500 ppm
I20260927 05:06:00.588037 2812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:00 dist-test-slave-hgs0 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:00.591105 2812 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:06:00.592669 2812 webserver.cc:460] Webserver started at http://127.0.97.62:33371/ using document root <none> and password file <none>
I20260927 05:06:00.593329 2812 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:00.593430 2812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:00.593688 2812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:00.596700 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "f71fce65c3124056b50838b8757e4058"
format_stamp: "Formatted at 2026-09-27 05:06:00 on dist-test-slave-hgs0"
server_key: "69bc2b7437476a2869a6466543251476"
server_key_iv: "cd1ac037b6271aeab73f5b95023126ee"
server_key_version: "encryptionkey@0"
I20260927 05:06:00.597417 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "f71fce65c3124056b50838b8757e4058"
format_stamp: "Formatted at 2026-09-27 05:06:00 on dist-test-slave-hgs0"
server_key: "69bc2b7437476a2869a6466543251476"
server_key_iv: "cd1ac037b6271aeab73f5b95023126ee"
server_key_version: "encryptionkey@0"
I20260927 05:06:00.602711 2812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 05:06:00.606360 2827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:00.608016 2812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 05:06:00.608204 2812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "f71fce65c3124056b50838b8757e4058"
format_stamp: "Formatted at 2026-09-27 05:06:00 on dist-test-slave-hgs0"
server_key: "69bc2b7437476a2869a6466543251476"
server_key_iv: "cd1ac037b6271aeab73f5b95023126ee"
server_key_version: "encryptionkey@0"
I20260927 05:06:00.608409 2812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:00.623828 2812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:00.627914 2812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:00.628257 2812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:00.646361 2812 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35893
I20260927 05:06:00.646391 2878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35893 every 8 connection(s)
I20260927 05:06:00.647691 2812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:00.651316 2879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:00.657487 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 2812
I20260927 05:06:00.657697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:00.658334 388 external_mini_cluster.cc:1428] Setting key 4396015e1d6d4002438c6c4f690f3e5c
I20260927 05:06:00.662137 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058: Bootstrap starting.
I20260927 05:06:00.665901 2879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:00.667294 2879 log.cc:824] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:00.670240 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058: No bootstrap required, opened a new log
Sep 27 05:06:00 dist-test-slave-hgs0 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:00.679912 2879 raft_consensus.cc:348] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71fce65c3124056b50838b8757e4058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35893 } }
I20260927 05:06:00.680419 2879 raft_consensus.cc:374] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:00.680536 2879 raft_consensus.cc:729] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71fce65c3124056b50838b8757e4058, State: Initialized, Role: FOLLOWER
I20260927 05:06:00.681252 2879 consensus_queue.cc:260] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71fce65c3124056b50838b8757e4058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35893 } }
I20260927 05:06:00.681535 2879 raft_consensus.cc:388] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:00.681689 2879 raft_consensus.cc:482] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:00.681881 2879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:00.683183 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:00.660457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.0.97.62:35893)
0927 05:06:00.661310 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:00.661325 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:00.661372 (+ 47us) server_negotiation.cc:407] Connection header received
0927 05:06:00.663749 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:00.663789 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:00.664342 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:00.664782 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:00.666050 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:00.668125 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:00.669502 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:00.670486 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:00.670696 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:00.674444 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:00.674461 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:00.674482 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:00.674547 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:00.678580 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:00.679243 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:00.679251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:00.679261 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:00.679449 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:00.679785 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:00.679790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:00.679793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:00.680539 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:00.680829 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:00.681256 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 05:06:00.681721 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":397,"threads_started":1}
I20260927 05:06:00.684821 2879 raft_consensus.cc:504] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71fce65c3124056b50838b8757e4058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35893 } }
I20260927 05:06:00.685338 2879 leader_election.cc:302] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f71fce65c3124056b50838b8757e4058; no voters:
I20260927 05:06:00.685976 2879 leader_election.cc:288] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:00.686094 2884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:00.686795 2884 raft_consensus.cc:686] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [term 1 LEADER]: Becoming Leader. State: Replica: f71fce65c3124056b50838b8757e4058, State: Running, Role: LEADER
I20260927 05:06:00.687378 2884 consensus_queue.cc:237] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71fce65c3124056b50838b8757e4058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35893 } }
I20260927 05:06:00.688279 2879 sys_catalog.cc:564] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:00.690168 2886 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f71fce65c3124056b50838b8757e4058. Latest consensus state: current_term: 1 leader_uuid: "f71fce65c3124056b50838b8757e4058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71fce65c3124056b50838b8757e4058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35893 } } }
I20260927 05:06:00.690619 2886 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:00.690279 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f71fce65c3124056b50838b8757e4058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71fce65c3124056b50838b8757e4058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35893 } } }
I20260927 05:06:00.691512 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:00.693188 2890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:00.698499 2890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:00.707736 2890 catalog_manager.cc:1347] Generated new cluster ID: b20b2c369a4c460fa50c6f03e8351c52
I20260927 05:06:00.707863 2890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:00.721760 2890 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:00.723280 2890 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:00.728744 2890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f71fce65c3124056b50838b8757e4058: Generated new TSK 0
I20260927 05:06:00.729533 2890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:06:00.783506 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35893
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:01.112880 2905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:01.113282 2905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:01.113438 2905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:01.120771 2905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:01.120949 2905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:01.121132 2905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:01.131238 2905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:01.132681 2905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:01.133991 2905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:01.147265 2910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:01.147289 2911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:01.147223 2913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:01.147921 2905 server_base.cc:1034] running on GCE node
I20260927 05:06:01.148562 2905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:01.149477 2905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:01.150707 2905 hybrid_clock.cc:648] HybridClock initialized: now 1790485561150649 us; error 50 us; skew 500 ppm
I20260927 05:06:01.151027 2905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:01 dist-test-slave-hgs0 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:01.154296 2905 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:06:01.155853 2905 webserver.cc:460] Webserver started at http://127.0.97.1:46111/ using document root <none> and password file <none>
I20260927 05:06:01.156591 2905 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:01.156704 2905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:01.157001 2905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:01.160089 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "9c7ec31dc5e04cc6a175eab7d4efce07"
format_stamp: "Formatted at 2026-09-27 05:06:01 on dist-test-slave-hgs0"
server_key: "d5ad52eb2e73abee22466902bd0bd51b"
server_key_iv: "12c68b312b43e554e78f422e934dfb73"
server_key_version: "encryptionkey@0"
I20260927 05:06:01.160923 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9c7ec31dc5e04cc6a175eab7d4efce07"
format_stamp: "Formatted at 2026-09-27 05:06:01 on dist-test-slave-hgs0"
server_key: "d5ad52eb2e73abee22466902bd0bd51b"
server_key_iv: "12c68b312b43e554e78f422e934dfb73"
server_key_version: "encryptionkey@0"
I20260927 05:06:01.166600 2905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:06:01.170100 2920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:01.171604 2905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:06:01.171787 2905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "9c7ec31dc5e04cc6a175eab7d4efce07"
format_stamp: "Formatted at 2026-09-27 05:06:01 on dist-test-slave-hgs0"
server_key: "d5ad52eb2e73abee22466902bd0bd51b"
server_key_iv: "12c68b312b43e554e78f422e934dfb73"
server_key_version: "encryptionkey@0"
I20260927 05:06:01.172060 2905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:01.202219 2905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:01.206461 2905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:01.206763 2905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:01.207754 2905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:01.209153 2905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:01.209228 2905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:01.209354 2905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:01.209414 2905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:01.243022 2905 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45473
I20260927 05:06:01.243161 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45473 every 8 connection(s)
I20260927 05:06:01.244411 2905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:01.254062 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 2905
I20260927 05:06:01.254310 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:06:01.254900 388 external_mini_cluster.cc:1428] Setting key ff8778c1045981c4086c43289721ff31
Sep 27 05:06:01 dist-test-slave-hgs0 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:01.266460 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:01.247158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48523 (local address 127.0.97.62:35893)
0927 05:06:01.247703 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:01.247709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:01.248839 (+ 1130us) server_negotiation.cc:407] Connection header received
0927 05:06:01.250958 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:01.250964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:01.251064 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:01.251188 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:01.252816 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:01.253535 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:01.255294 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:01.255700 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:01.255840 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:01.259619 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:01.259629 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:01.259635 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:01.259681 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:01.262771 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:01.263846 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:01.263850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:01.263853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:01.264035 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:01.264569 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:01.264575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:01.264579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:01.264859 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:01.264989 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:01.265996 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0927 05:06:01.266201 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":274,"threads_started":1}
I20260927 05:06:01.268085 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:01.247837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35893 (local address 127.0.97.1:48523)
0927 05:06:01.248661 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:01.248700 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:01.250655 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:01.251431 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:01.251440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:01.252032 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:01.252639 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:01.252657 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:01.253719 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:01.253723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:01.255144 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:01.255156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:01.255932 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:01.255964 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:01.256220 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:01.256914 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:01.256936 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:01.259367 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:01.262950 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:01.262963 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:01.262986 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:01.263700 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:01.264143 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:01.264150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:01.264157 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:01.264413 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:01.265005 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:01.265016 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:01.265523 (+ 507us) client_negotiation.cc:769] Sending connection context
0927 05:06:01.265860 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 05:06:01.266479 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":321,"threads_started":1}
I20260927 05:06:01.269977 3033 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35893
I20260927 05:06:01.270373 3033 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:01.271191 3033 heartbeater.cc:511] Master 127.0.97.62:35893 requested a full tablet report, sending...
I20260927 05:06:01.273514 2844 ts_manager.cc:194] Registered new tserver with Master: 9c7ec31dc5e04cc6a175eab7d4efce07 (127.0.97.1:45473)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:06:01.275306 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:06:01.303874 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35893
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:01.635928 3039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:01.636373 3039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:01.636539 3039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:01.643800 3039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:01.643996 3039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:01.644182 3039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:01.654201 3039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:01.655508 3039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:01.656895 3039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:01.670190 3047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:01.670217 3044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:01.670296 3045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:01.671010 3039 server_base.cc:1034] running on GCE node
I20260927 05:06:01.671617 3039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:01.672681 3039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:01.673913 3039 hybrid_clock.cc:648] HybridClock initialized: now 1790485561673866 us; error 47 us; skew 500 ppm
I20260927 05:06:01.674247 3039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:01 dist-test-slave-hgs0 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:01.677565 3039 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:06:01.679127 3039 webserver.cc:460] Webserver started at http://127.0.97.2:35655/ using document root <none> and password file <none>
I20260927 05:06:01.679842 3039 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:01.680013 3039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:01.680392 3039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:01.683348 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "8b52bdbf8f34437787f11b405472c5fb"
format_stamp: "Formatted at 2026-09-27 05:06:01 on dist-test-slave-hgs0"
server_key: "db7a01566d25a57731cddd20af87b288"
server_key_iv: "f0254275dc915f17984df0b34baa5384"
server_key_version: "encryptionkey@0"
I20260927 05:06:01.684242 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8b52bdbf8f34437787f11b405472c5fb"
format_stamp: "Formatted at 2026-09-27 05:06:01 on dist-test-slave-hgs0"
server_key: "db7a01566d25a57731cddd20af87b288"
server_key_iv: "f0254275dc915f17984df0b34baa5384"
server_key_version: "encryptionkey@0"
I20260927 05:06:01.689808 3039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:06:01.693715 3054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:01.695261 3039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:06:01.695462 3039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "8b52bdbf8f34437787f11b405472c5fb"
format_stamp: "Formatted at 2026-09-27 05:06:01 on dist-test-slave-hgs0"
server_key: "db7a01566d25a57731cddd20af87b288"
server_key_iv: "f0254275dc915f17984df0b34baa5384"
server_key_version: "encryptionkey@0"
I20260927 05:06:01.695700 3039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:01.709566 3039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:01.714226 3039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:01.714537 3039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:01.715503 3039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:01.716969 3039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:01.717074 3039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:01.717233 3039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:01.717304 3039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:01.752094 3039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41587
I20260927 05:06:01.752110 3166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41587 every 8 connection(s)
I20260927 05:06:01.753422 3039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:01.763703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 3039
I20260927 05:06:01.763907 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:06:01.764469 388 external_mini_cluster.cc:1428] Setting key f1502b7c470f8f5d1be7f70a85ad98a2
Sep 27 05:06:01 dist-test-slave-hgs0 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:01.775246 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:01.756085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46379 (local address 127.0.97.62:35893)
0927 05:06:01.756262 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:01.756266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:01.757535 (+ 1269us) server_negotiation.cc:407] Connection header received
0927 05:06:01.759346 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:01.759350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:01.759419 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:01.759521 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:01.761233 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:01.761949 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:01.763593 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:01.764283 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:01.764426 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:01.768571 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:01.768577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:01.768581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:01.768617 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:01.771513 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:01.772512 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:01.772517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:01.772520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:01.772676 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:01.773195 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:01.773200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:01.773206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:01.773631 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:01.773823 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:01.774628 (+ 805us) server_negotiation.cc:299] Negotiation successful
0927 05:06:01.774882 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:06:01.776463 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:01.756660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35893 (local address 127.0.97.2:46379)
0927 05:06:01.757371 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:01.757410 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:01.759034 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:01.759797 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:01.759807 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:01.760468 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:01.761043 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:01.761070 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:01.762130 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:01.762134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:01.763425 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:01.763442 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:01.764511 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:01.764517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:01.764812 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:01.765597 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:01.765623 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:01.768294 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:01.771712 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:01.771724 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:01.771746 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:01.772346 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:01.772810 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:01.772816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:01.772820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:01.773058 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:01.773794 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:01.773803 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:01.774195 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 05:06:01.774488 (+ 293us) client_negotiation.cc:240] Negotiation successful
0927 05:06:01.775096 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":276,"threads_started":1}
I20260927 05:06:01.778187 3167 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35893
I20260927 05:06:01.778561 3167 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:01.779398 3167 heartbeater.cc:511] Master 127.0.97.62:35893 requested a full tablet report, sending...
I20260927 05:06:01.781225 2844 ts_manager.cc:194] Registered new tserver with Master: 8b52bdbf8f34437787f11b405472c5fb (127.0.97.2:41587)
I20260927 05:06:01.782624 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46379
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:06:01.812772 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35893
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:02.161264 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:02.161561 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:02.161664 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:02.169005 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:02.169138 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:02.169255 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:02.180773 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:02.182154 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:02.183486 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:02.198182 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:02.198205 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:02.198287 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:02.199791 3172 server_base.cc:1034] running on GCE node
I20260927 05:06:02.200764 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:02.202111 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:02.203485 3172 hybrid_clock.cc:648] HybridClock initialized: now 1790485562203387 us; error 84 us; skew 500 ppm
I20260927 05:06:02.203912 3172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:02 dist-test-slave-hgs0 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:02.208415 3172 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:06:02.210287 3172 webserver.cc:460] Webserver started at http://127.0.97.3:32887/ using document root <none> and password file <none>
I20260927 05:06:02.211097 3172 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:02.211258 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:02.211678 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:02.215013 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "8aa6e3a20149411cb61ed758002723b0"
format_stamp: "Formatted at 2026-09-27 05:06:02 on dist-test-slave-hgs0"
server_key: "7dbd32ff88edf71322ee2f18a616a329"
server_key_iv: "035b62c6f009dfeba6a7622a87494b2b"
server_key_version: "encryptionkey@0"
I20260927 05:06:02.215912 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8aa6e3a20149411cb61ed758002723b0"
format_stamp: "Formatted at 2026-09-27 05:06:02 on dist-test-slave-hgs0"
server_key: "7dbd32ff88edf71322ee2f18a616a329"
server_key_iv: "035b62c6f009dfeba6a7622a87494b2b"
server_key_version: "encryptionkey@0"
I20260927 05:06:02.221863 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:06:02.227116 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:02.229281 3172 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 05:06:02.229519 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "8aa6e3a20149411cb61ed758002723b0"
format_stamp: "Formatted at 2026-09-27 05:06:02 on dist-test-slave-hgs0"
server_key: "7dbd32ff88edf71322ee2f18a616a329"
server_key_iv: "035b62c6f009dfeba6a7622a87494b2b"
server_key_version: "encryptionkey@0"
I20260927 05:06:02.229806 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:02.247280 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:02.252602 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:02.253294 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:02.254626 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:02.256345 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:02.256459 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:02.256634 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:02.256717 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:02.279477 3033 heartbeater.cc:503] Master 127.0.97.62:35893 was elected leader, sending a full tablet report...
I20260927 05:06:02.296763 3172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35415
I20260927 05:06:02.296861 3299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35415 every 8 connection(s)
I20260927 05:06:02.298286 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:02.303715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 3172
I20260927 05:06:02.303926 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:06:02.304513 388 external_mini_cluster.cc:1428] Setting key 579718d5a2c7dd3908c405328c3c8903
Sep 27 05:06:02 dist-test-slave-hgs0 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:02.323738 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:02.301508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40483 (local address 127.0.97.62:35893)
0927 05:06:02.302068 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:02.302074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:02.303281 (+ 1207us) server_negotiation.cc:407] Connection header received
0927 05:06:02.305979 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:02.305983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:02.306136 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:02.306265 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:02.308551 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:02.309421 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:02.311240 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:02.311741 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:02.311976 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:02.316234 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:02.316241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:02.316245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:02.316285 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:02.319597 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:02.321009 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:02.321014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:02.321018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:02.321193 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:02.321927 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:02.321931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:02.321933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:02.322249 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:02.322373 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:02.323235 (+ 862us) server_negotiation.cc:299] Negotiation successful
0927 05:06:02.323432 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":294,"threads_started":1}
I20260927 05:06:02.324949 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:02.302212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35893 (local address 127.0.97.3:40483)
0927 05:06:02.303063 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:02.303106 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:02.305645 (+ 2539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:02.306690 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:02.306701 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:02.307409 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:02.308318 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:02.308349 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:02.309653 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:02.309658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:02.311069 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:02.311083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:02.312081 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:02.312089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:02.312327 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:02.313285 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:02.313317 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:02.315878 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:02.319853 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:02.319869 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:02.319894 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:02.320842 (+ 948us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:02.321381 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:02.321390 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:02.321397 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:02.321789 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:02.322378 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:02.322387 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:02.322841 (+ 454us) client_negotiation.cc:769] Sending connection context
0927 05:06:02.323110 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 05:06:02.323593 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":384,"threads_started":1}
I20260927 05:06:02.326893 3300 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35893
I20260927 05:06:02.327335 3300 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:02.328356 3300 heartbeater.cc:511] Master 127.0.97.62:35893 requested a full tablet report, sending...
I20260927 05:06:02.330302 2844 ts_manager.cc:194] Registered new tserver with Master: 8aa6e3a20149411cb61ed758002723b0 (127.0.97.3:35415)
I20260927 05:06:02.331369 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40483
I20260927 05:06:02.334751 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 05:06:02.367645 3302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:02.364847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.0.97.62:35893)
0927 05:06:02.365077 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:02.365082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:02.365583 (+ 501us) server_negotiation.cc:407] Connection header received
0927 05:06:02.365749 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:02.365753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:02.365832 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:02.365936 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:02.367325 (+ 1389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49538: BlockingRecv error: recv got EOF from 127.0.0.1:49538 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:06:02.371145 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 2905
W20260927 05:06:02.374712 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 2905: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:45473: server requires authentication, but client does not have Kerberos credentials available
I20260927 05:06:02.374967 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 2905
I20260927 05:06:02.391426 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3039
W20260927 05:06:02.394801 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3039: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:41587: server requires authentication, but client does not have Kerberos credentials available
I20260927 05:06:02.394989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 3039
I20260927 05:06:02.411090 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3172
W20260927 05:06:02.414471 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3172: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:35415: server requires authentication, but client does not have Kerberos credentials available
I20260927 05:06:02.414644 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 3172
I20260927 05:06:02.430754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 2812
I20260927 05:06:02.432228 2874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:02.540942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 2812
2026-09-27T05:06:02Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3577 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:06:02 dist-test-slave-hgs0 krb5kdc[3318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:06:02 dist-test-slave-hgs0 krb5kdc[3318](info): set up 2 sockets
Sep 27 05:06:02 dist-test-slave-hgs0 krb5kdc[3318](info): commencing operation
krb5kdc: starting...
W20260927 05:06:03.684540 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:06:03 dist-test-slave-hgs0 krb5kdc[3318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:06:03.896275 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37015
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:04.253841 3331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:04.254182 3331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:04.254391 3331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:04.262168 3331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:04.262295 3331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:04.262374 3331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:04.262478 3331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:04.262562 3331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:04.273767 3331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:04.275463 3331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:04.277434 3331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:04.291168 3337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:04.291229 3336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:04.291308 3339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:04.292264 3331 server_base.cc:1034] running on GCE node
I20260927 05:06:04.293262 3331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:04.295377 3331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:04.296702 3331 hybrid_clock.cc:648] HybridClock initialized: now 1790485564296636 us; error 78 us; skew 500 ppm
I20260927 05:06:04.297192 3331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:04 dist-test-slave-hgs0 krb5kdc[3318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:04.305245 3331 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:06:04.307111 3331 webserver.cc:460] Webserver started at http://127.0.97.62:33363/ using document root <none> and password file <none>
I20260927 05:06:04.308262 3331 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:04.308430 3331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:04.308779 3331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:04.312285 3331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "57e6ee6371ec4b18b045c8d2b66117fb"
format_stamp: "Formatted at 2026-09-27 05:06:04 on dist-test-slave-hgs0"
server_key: "b908d89292162b952219b6249d3653e5"
server_key_iv: "b658d205c9d2406952c3468267fabbeb"
server_key_version: "encryptionkey@0"
I20260927 05:06:04.313262 3331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "57e6ee6371ec4b18b045c8d2b66117fb"
format_stamp: "Formatted at 2026-09-27 05:06:04 on dist-test-slave-hgs0"
server_key: "b908d89292162b952219b6249d3653e5"
server_key_iv: "b658d205c9d2406952c3468267fabbeb"
server_key_version: "encryptionkey@0"
I20260927 05:06:04.319762 3331 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 05:06:04.324235 3346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:04.325965 3331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:06:04.326191 3331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "57e6ee6371ec4b18b045c8d2b66117fb"
format_stamp: "Formatted at 2026-09-27 05:06:04 on dist-test-slave-hgs0"
server_key: "b908d89292162b952219b6249d3653e5"
server_key_iv: "b658d205c9d2406952c3468267fabbeb"
server_key_version: "encryptionkey@0"
I20260927 05:06:04.326442 3331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:04.350467 3331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:04.355288 3331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:04.355638 3331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:04.376413 3331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37015
I20260927 05:06:04.376434 3397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37015 every 8 connection(s)
I20260927 05:06:04.377907 3331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:04.381716 3398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:04.388046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 3331
I20260927 05:06:04.388298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:04.389094 388 external_mini_cluster.cc:1428] Setting key 9322f2b8b83c01bf08339c0eb71c79cf
I20260927 05:06:04.393492 3398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb: Bootstrap starting.
I20260927 05:06:04.397631 3398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:04.398962 3398 log.cc:824] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:04.402146 3398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb: No bootstrap required, opened a new log
Sep 27 05:06:04 dist-test-slave-hgs0 krb5kdc[3318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:04.410138 3398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6ee6371ec4b18b045c8d2b66117fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } }
I20260927 05:06:04.410606 3398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:04.410738 3398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57e6ee6371ec4b18b045c8d2b66117fb, State: Initialized, Role: FOLLOWER
I20260927 05:06:04.411530 3398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6ee6371ec4b18b045c8d2b66117fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } }
I20260927 05:06:04.411841 3398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:04.412076 3398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:04.412292 3398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:04.413255 3401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:04.391206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.0.97.62:37015)
0927 05:06:04.392096 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:04.392109 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:04.392155 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:06:04.393969 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:04.393998 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:04.394428 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:04.394889 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:04.396772 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:04.398302 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:04.400218 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:04.400977 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:04.401206 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:04.405321 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:04.405330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:04.405351 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:04.405397 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:04.408710 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:04.409470 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:04.409477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:04.409485 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:04.409661 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:04.410061 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:04.410065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:04.410067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:04.410729 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:04.411042 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:04.411498 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 05:06:04.411937 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":383,"threads_started":1}
I20260927 05:06:04.415125 3398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6ee6371ec4b18b045c8d2b66117fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } }
I20260927 05:06:04.415794 3398 leader_election.cc:302] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57e6ee6371ec4b18b045c8d2b66117fb; no voters:
I20260927 05:06:04.416644 3398 leader_election.cc:288] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:04.416770 3403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:04.418017 3403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [term 1 LEADER]: Becoming Leader. State: Replica: 57e6ee6371ec4b18b045c8d2b66117fb, State: Running, Role: LEADER
I20260927 05:06:04.418617 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:06:04.418777 3403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6ee6371ec4b18b045c8d2b66117fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } }
I20260927 05:06:04.419312 3398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:04.426362 3405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57e6ee6371ec4b18b045c8d2b66117fb. Latest consensus state: current_term: 1 leader_uuid: "57e6ee6371ec4b18b045c8d2b66117fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6ee6371ec4b18b045c8d2b66117fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } }
I20260927 05:06:04.426793 3405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:04.427372 3404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57e6ee6371ec4b18b045c8d2b66117fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57e6ee6371ec4b18b045c8d2b66117fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37015 } } }
I20260927 05:06:04.427688 3404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb [sys.catalog]: This master's current role is: LEADER
W20260927 05:06:04.443818 3417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:06:04.444020 3417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:06:04.444219 3418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:04.446511 3418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:04.455086 3418 catalog_manager.cc:1347] Generated new cluster ID: a4241a2c2f9e4769a10b74af4912f657
I20260927 05:06:04.455209 3418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:04.474059 3401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:04.465987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.0.97.62:37015)
0927 05:06:04.466231 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:04.466245 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:04.466858 (+ 613us) server_negotiation.cc:407] Connection header received
0927 05:06:04.467074 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:04.467084 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:04.467162 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:04.467284 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:04.468345 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:04.469127 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:04.470482 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:04.470954 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:04.471109 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:04.472378 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:04.472383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:04.472387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:04.473096 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:04.473178 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:04.473210 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:04.473346 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:04.473494 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 05:06:04.473612 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 05:06:04.492169 3418 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:04.493762 3418 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:04.501374 3418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57e6ee6371ec4b18b045c8d2b66117fb: Generated new TSK 0
I20260927 05:06:04.502645 3418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:06:04.513636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3331
I20260927 05:06:04.515839 3393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:04.616056 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 3331
2026-09-27T05:06:04Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2067 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:06:04 dist-test-slave-hgs0 krb5kdc[3432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:06:04 dist-test-slave-hgs0 krb5kdc[3432](info): set up 2 sockets
Sep 27 05:06:04 dist-test-slave-hgs0 krb5kdc[3432](info): commencing operation
krb5kdc: starting...
W20260927 05:06:05.720836 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:06:05 dist-test-slave-hgs0 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:06:05.932473 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36893
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:06.276993 3445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:06.277395 3445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:06.277570 3445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:06.285758 3445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:06.285935 3445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:06.286049 3445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:06.286146 3445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:06.286232 3445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:06.296586 3445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:06.298079 3445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:06.299427 3445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:06.311832 3451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:06.311836 3453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:06.312675 3450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:06.313442 3445 server_base.cc:1034] running on GCE node
I20260927 05:06:06.314349 3445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:06.316001 3445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:06.317227 3445 hybrid_clock.cc:648] HybridClock initialized: now 1790485566317166 us; error 59 us; skew 500 ppm
I20260927 05:06:06.317584 3445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:06 dist-test-slave-hgs0 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:06.320945 3445 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:06:06.322496 3445 webserver.cc:460] Webserver started at http://127.0.97.62:37017/ using document root <none> and password file <none>
I20260927 05:06:06.323204 3445 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:06.323342 3445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:06.323660 3445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:06.326807 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "e30fd40bb03b442297942b050df4ca4d"
format_stamp: "Formatted at 2026-09-27 05:06:06 on dist-test-slave-hgs0"
server_key: "969bd2ec569d2d2cf758167de5954a31"
server_key_iv: "7ccf99beea59d0c3a09afa1764f53d76"
server_key_version: "encryptionkey@0"
I20260927 05:06:06.327605 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "e30fd40bb03b442297942b050df4ca4d"
format_stamp: "Formatted at 2026-09-27 05:06:06 on dist-test-slave-hgs0"
server_key: "969bd2ec569d2d2cf758167de5954a31"
server_key_iv: "7ccf99beea59d0c3a09afa1764f53d76"
server_key_version: "encryptionkey@0"
I20260927 05:06:06.333096 3445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:06:06.336989 3460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:06.338557 3445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:06:06.338793 3445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "e30fd40bb03b442297942b050df4ca4d"
format_stamp: "Formatted at 2026-09-27 05:06:06 on dist-test-slave-hgs0"
server_key: "969bd2ec569d2d2cf758167de5954a31"
server_key_iv: "7ccf99beea59d0c3a09afa1764f53d76"
server_key_version: "encryptionkey@0"
I20260927 05:06:06.339026 3445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:06.355537 3445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:06.360080 3445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:06.360405 3445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:06.379900 3445 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36893
I20260927 05:06:06.379992 3511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36893 every 8 connection(s)
I20260927 05:06:06.381438 3445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:06.383016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 3445
I20260927 05:06:06.383242 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:06.383859 388 external_mini_cluster.cc:1428] Setting key bcb1f8c67cb70706dd723c57cfbf601b
I20260927 05:06:06.386231 3512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 05:06:06 dist-test-slave-hgs0 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:06.401870 3512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d: Bootstrap starting.
I20260927 05:06:06.406580 3512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:06.407907 3512 log.cc:824] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:06.411907 3512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d: No bootstrap required, opened a new log
I20260927 05:06:06.411959 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:06.386095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.0.97.62:36893)
0927 05:06:06.387007 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:06.387026 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:06.387082 (+ 56us) server_negotiation.cc:407] Connection header received
0927 05:06:06.389622 (+ 2540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:06.389658 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:06.390314 (+ 656us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:06.390854 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:06.392409 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:06.394740 (+ 2331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:06.396203 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:06.397397 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:06.397620 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:06.402089 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:06.402102 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:06.402123 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:06.402195 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:06.406629 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:06.407505 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:06.407522 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:06.407534 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:06.407764 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:06.408306 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:06.408311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:06.408315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:06.409164 (+ 849us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:06.409438 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:06.410067 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 05:06:06.410544 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":387,"threads_started":1}
I20260927 05:06:06.419610 3512 raft_consensus.cc:348] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30fd40bb03b442297942b050df4ca4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36893 } }
I20260927 05:06:06.420095 3512 raft_consensus.cc:374] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:06.420184 3512 raft_consensus.cc:729] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30fd40bb03b442297942b050df4ca4d, State: Initialized, Role: FOLLOWER
I20260927 05:06:06.420744 3512 consensus_queue.cc:260] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30fd40bb03b442297942b050df4ca4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36893 } }
I20260927 05:06:06.420953 3512 raft_consensus.cc:388] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:06.421141 3512 raft_consensus.cc:482] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:06.421336 3512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:06.423722 3512 raft_consensus.cc:504] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30fd40bb03b442297942b050df4ca4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36893 } }
I20260927 05:06:06.424317 3512 leader_election.cc:302] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e30fd40bb03b442297942b050df4ca4d; no voters:
I20260927 05:06:06.424999 3512 leader_election.cc:288] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:06.425141 3517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:06.425916 3517 raft_consensus.cc:686] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [term 1 LEADER]: Becoming Leader. State: Replica: e30fd40bb03b442297942b050df4ca4d, State: Running, Role: LEADER
I20260927 05:06:06.426657 3517 consensus_queue.cc:237] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30fd40bb03b442297942b050df4ca4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36893 } }
I20260927 05:06:06.427604 3512 sys_catalog.cc:564] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:06.430048 3518 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e30fd40bb03b442297942b050df4ca4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30fd40bb03b442297942b050df4ca4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36893 } } }
I20260927 05:06:06.430039 3519 sys_catalog.cc:455] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e30fd40bb03b442297942b050df4ca4d. Latest consensus state: current_term: 1 leader_uuid: "e30fd40bb03b442297942b050df4ca4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e30fd40bb03b442297942b050df4ca4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36893 } } }
I20260927 05:06:06.430477 3518 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:06.430477 3519 sys_catalog.cc:458] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:06.432729 3523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:06.438944 3523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:06.448935 3523 catalog_manager.cc:1347] Generated new cluster ID: c32e6e4ee893440cbbcf4f3b24fc010f
I20260927 05:06:06.449074 3523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:06.463238 3523 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:06.464895 3523 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:06.477535 3523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e30fd40bb03b442297942b050df4ca4d: Generated new TSK 0
I20260927 05:06:06.478601 3523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:06:06.539529 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36893
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:06.918682 3538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:06.918982 3538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:06.919128 3538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:06.926746 3538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:06.926875 3538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:06.927044 3538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:06.937613 3538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:06.938987 3538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:06.940426 3538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:06.956275 3544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:06.956357 3543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:06.956337 3546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:06.957566 3538 server_base.cc:1034] running on GCE node
I20260927 05:06:06.958250 3538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:06.959254 3538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:06.960506 3538 hybrid_clock.cc:648] HybridClock initialized: now 1790485566960424 us; error 66 us; skew 500 ppm
I20260927 05:06:06.960935 3538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:06 dist-test-slave-hgs0 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:06.964646 3538 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:06:06.966408 3538 webserver.cc:460] Webserver started at http://127.0.97.1:45045/ using document root <none> and password file <none>
I20260927 05:06:06.967190 3538 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:06.967303 3538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:06.967584 3538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:06.970965 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "82543819428a4e38b3b7e0ba1e1c93b2"
format_stamp: "Formatted at 2026-09-27 05:06:06 on dist-test-slave-hgs0"
server_key: "4a244bde698cd6707c2c4a66f6734e36"
server_key_iv: "b75b33c770218957c562e75a6f5742bc"
server_key_version: "encryptionkey@0"
I20260927 05:06:06.971836 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "82543819428a4e38b3b7e0ba1e1c93b2"
format_stamp: "Formatted at 2026-09-27 05:06:06 on dist-test-slave-hgs0"
server_key: "4a244bde698cd6707c2c4a66f6734e36"
server_key_iv: "b75b33c770218957c562e75a6f5742bc"
server_key_version: "encryptionkey@0"
I20260927 05:06:06.977918 3538 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 05:06:06.981999 3553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:06.983628 3538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:06:06.983862 3538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "82543819428a4e38b3b7e0ba1e1c93b2"
format_stamp: "Formatted at 2026-09-27 05:06:06 on dist-test-slave-hgs0"
server_key: "4a244bde698cd6707c2c4a66f6734e36"
server_key_iv: "b75b33c770218957c562e75a6f5742bc"
server_key_version: "encryptionkey@0"
I20260927 05:06:06.984171 3538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:07.004287 3538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:07.010979 3538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:07.011371 3538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:07.012833 3538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:07.014689 3538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:07.014777 3538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:07.014935 3538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:07.015004 3538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:07.051651 3538 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42397
I20260927 05:06:07.051739 3665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42397 every 8 connection(s)
I20260927 05:06:07.053195 3538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:07.053876 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 3538
I20260927 05:06:07.054105 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:06:07.054961 388 external_mini_cluster.cc:1428] Setting key 600e61f443a6fc5a5606604cdc59641c
Sep 27 05:06:07 dist-test-slave-hgs0 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:06:07.079017 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:07.056204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40645 (local address 127.0.97.62:36893)
0927 05:06:07.056695 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:07.056702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:07.057925 (+ 1223us) server_negotiation.cc:407] Connection header received
0927 05:06:07.059792 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:07.059801 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:07.059910 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:07.060079 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:07.061692 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:07.062642 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:07.065608 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:07.066068 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:07.066208 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:07.071095 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:07.071106 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:07.071115 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:07.071186 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:07.075392 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:07.076476 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:07.076485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:07.076492 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:07.076669 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:07.077125 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:07.077129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:07.077131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:07.077502 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:07.077621 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:07.078432 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 05:06:07.078654 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:06:07.080392 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:07.056896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36893 (local address 127.0.97.1:40645)
0927 05:06:07.057740 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:07.057795 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:07.059492 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:07.060299 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:07.060307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:07.060879 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:07.061489 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:07.061508 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:07.062927 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:07.062936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:07.065399 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:07.065420 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:07.066349 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:07.066359 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:07.066718 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:07.067566 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:07.067602 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:07.070704 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:07.075600 (+ 4896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:07.075610 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:07.075627 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:07.076273 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:07.076785 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:07.076788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:07.076791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:07.077016 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:07.077632 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:07.077639 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:07.078041 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 05:06:07.078298 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 05:06:07.078786 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":349,"threads_started":1}
I20260927 05:06:07.082263 3666 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36893
I20260927 05:06:07.082581 3666 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:07.083362 3666 heartbeater.cc:511] Master 127.0.97.62:36893 requested a full tablet report, sending...
I20260927 05:06:07.085603 3477 ts_manager.cc:194] Registered new tserver with Master: 82543819428a4e38b3b7e0ba1e1c93b2 (127.0.97.1:42397)
I20260927 05:06:07.087114 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:06:07.107067 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36893
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:07.473774 3672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:07.474102 3672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:07.474282 3672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:07.482429 3672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:07.482568 3672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:07.482784 3672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:07.494243 3672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:07.495754 3672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:07.497349 3672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:07.512442 3677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:07.512485 3678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:07.512923 3680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:07.513556 3672 server_base.cc:1034] running on GCE node
I20260927 05:06:07.514283 3672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:07.515337 3672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:07.516552 3672 hybrid_clock.cc:648] HybridClock initialized: now 1790485567516502 us; error 42 us; skew 500 ppm
I20260927 05:06:07.516935 3672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:07 dist-test-slave-hgs0 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:07.520720 3672 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:06:07.522578 3672 webserver.cc:460] Webserver started at http://127.0.97.2:37485/ using document root <none> and password file <none>
I20260927 05:06:07.523387 3672 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:07.523500 3672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:07.523859 3672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:07.527209 3672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "e19152cdd1bb4c67a9fa3a2207a2fbb7"
format_stamp: "Formatted at 2026-09-27 05:06:07 on dist-test-slave-hgs0"
server_key: "72042094918aa1fb619442d0dc33b7e0"
server_key_iv: "a9ec35981c443058cb24601578326dfd"
server_key_version: "encryptionkey@0"
I20260927 05:06:07.528190 3672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e19152cdd1bb4c67a9fa3a2207a2fbb7"
format_stamp: "Formatted at 2026-09-27 05:06:07 on dist-test-slave-hgs0"
server_key: "72042094918aa1fb619442d0dc33b7e0"
server_key_iv: "a9ec35981c443058cb24601578326dfd"
server_key_version: "encryptionkey@0"
I20260927 05:06:07.534440 3672 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:06:07.538568 3687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:07.540256 3672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:07.540454 3672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "e19152cdd1bb4c67a9fa3a2207a2fbb7"
format_stamp: "Formatted at 2026-09-27 05:06:07 on dist-test-slave-hgs0"
server_key: "72042094918aa1fb619442d0dc33b7e0"
server_key_iv: "a9ec35981c443058cb24601578326dfd"
server_key_version: "encryptionkey@0"
I20260927 05:06:07.540730 3672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:07.566855 3672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:07.572367 3672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:07.572801 3672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:07.574225 3672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:07.576125 3672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:07.576215 3672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:07.576438 3672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:07.576512 3672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:07.615075 3672 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46161
I20260927 05:06:07.615289 3799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46161 every 8 connection(s)
I20260927 05:06:07.616602 3672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:07.619153 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 3672
I20260927 05:06:07.619367 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:06:07.620085 388 external_mini_cluster.cc:1428] Setting key 582e0abebba08bd14bbe68faf6199dca
Sep 27 05:06:07 dist-test-slave-hgs0 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:06:07.646420 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:07.619996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50543 (local address 127.0.97.62:36893)
0927 05:06:07.624317 (+ 4321us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:07.624323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:07.624346 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:06:07.626975 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:07.626981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:07.627068 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:07.627184 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:07.629977 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:07.631296 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:07.633229 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:07.634047 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:07.634238 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:07.638772 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:07.638784 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:07.638793 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:07.638855 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:07.641991 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:07.643335 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:07.643340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:07.643343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:07.643543 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:07.644235 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:07.644239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:07.644241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:07.644697 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:07.644859 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:07.645821 (+ 962us) server_negotiation.cc:299] Negotiation successful
0927 05:06:07.646101 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4109,"thread_start_us":3969,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:06:07.647840 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:07.620904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36893 (local address 127.0.97.2:50543)
0927 05:06:07.622992 (+ 2088us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:07.623074 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:07.626496 (+ 3422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:07.627578 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:07.627596 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:07.628655 (+ 1059us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:07.629707 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:07.629746 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:07.631526 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:07.631531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:07.633007 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:07.633048 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:07.634400 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:07.634407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:07.634648 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:07.635487 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:07.635515 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:07.638408 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:07.642208 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:07.642221 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:07.642246 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:07.643104 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:07.643720 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:07.643729 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:07.643736 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:07.644082 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:07.644861 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:07.644869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:07.645326 (+ 457us) client_negotiation.cc:769] Sending connection context
0927 05:06:07.645722 (+ 396us) client_negotiation.cc:240] Negotiation successful
0927 05:06:07.646387 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1660,"thread_start_us":1312,"threads_started":1}
I20260927 05:06:07.649912 3800 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36893
I20260927 05:06:07.650349 3800 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:07.651443 3800 heartbeater.cc:511] Master 127.0.97.62:36893 requested a full tablet report, sending...
I20260927 05:06:07.653483 3477 ts_manager.cc:194] Registered new tserver with Master: e19152cdd1bb4c67a9fa3a2207a2fbb7 (127.0.97.2:46161)
I20260927 05:06:07.654433 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:06:07.675767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36893
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:08.030665 3806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:08.030959 3806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:08.031111 3806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:08.038757 3806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:08.038872 3806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:08.039045 3806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:08.049345 3806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:08.050747 3806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:08.052330 3806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:08.066331 3812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:08.066454 3811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:08.066893 3814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:08.067584 3806 server_base.cc:1034] running on GCE node
I20260927 05:06:08.068457 3806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:08.069581 3806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:08.070854 3806 hybrid_clock.cc:648] HybridClock initialized: now 1790485568070781 us; error 63 us; skew 500 ppm
I20260927 05:06:08.071245 3806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:08 dist-test-slave-hgs0 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:08.075137 3806 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:06:08.076886 3806 webserver.cc:460] Webserver started at http://127.0.97.3:45557/ using document root <none> and password file <none>
I20260927 05:06:08.077605 3806 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:08.077737 3806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:08.078011 3806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:08.081152 3806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "297c6660df9a49ada87d711cd36c478b"
format_stamp: "Formatted at 2026-09-27 05:06:08 on dist-test-slave-hgs0"
server_key: "2ae802a5606dce83f3f67e47b58dcbbb"
server_key_iv: "88441b2cf38e5bf41127dfbf4c0fa3e8"
server_key_version: "encryptionkey@0"
I20260927 05:06:08.081916 3806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "297c6660df9a49ada87d711cd36c478b"
format_stamp: "Formatted at 2026-09-27 05:06:08 on dist-test-slave-hgs0"
server_key: "2ae802a5606dce83f3f67e47b58dcbbb"
server_key_iv: "88441b2cf38e5bf41127dfbf4c0fa3e8"
server_key_version: "encryptionkey@0"
I20260927 05:06:08.087211 3806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:06:08.090570 3666 heartbeater.cc:503] Master 127.0.97.62:36893 was elected leader, sending a full tablet report...
I20260927 05:06:08.091012 3821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:08.092743 3806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:08.092880 3806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "297c6660df9a49ada87d711cd36c478b"
format_stamp: "Formatted at 2026-09-27 05:06:08 on dist-test-slave-hgs0"
server_key: "2ae802a5606dce83f3f67e47b58dcbbb"
server_key_iv: "88441b2cf38e5bf41127dfbf4c0fa3e8"
server_key_version: "encryptionkey@0"
I20260927 05:06:08.093045 3806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:08.151154 3806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:08.155897 3806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:08.156333 3806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:08.157395 3806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:08.158861 3806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:08.158936 3806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:08.159121 3806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:08.159174 3806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:08.194924 3806 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32853
I20260927 05:06:08.195044 3933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32853 every 8 connection(s)
I20260927 05:06:08.196367 3806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:08.197952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 3806
I20260927 05:06:08.198128 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:06:08.198596 388 external_mini_cluster.cc:1428] Setting key 00c2288f4a47e4a9d9dc546d9fa7e191
Sep 27 05:06:08 dist-test-slave-hgs0 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:08.218670 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:08.199417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57811 (local address 127.0.97.62:36893)
0927 05:06:08.200296 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:08.200303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:08.201226 (+ 923us) server_negotiation.cc:407] Connection header received
0927 05:06:08.203251 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:08.203255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:08.203330 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:08.203425 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:08.205185 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.206043 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.207763 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.208222 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.208320 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:08.211986 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:08.212006 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:08.212010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:08.212043 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:08.215277 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:08.216122 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:08.216129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:08.216134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:08.216266 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:08.216751 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:08.216757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:08.216761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:08.217046 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:08.217225 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:08.218008 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 05:06:08.218246 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":571,"threads_started":1}
I20260927 05:06:08.219749 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:08.200098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36893 (local address 127.0.97.3:57811)
0927 05:06:08.201040 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:08.201110 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:08.202968 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:08.203655 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:08.203664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:08.204383 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:08.205008 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:08.205027 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.206213 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.206218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:08.207632 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:08.207645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.208425 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.208431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:08.208662 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:08.209416 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:08.209440 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:08.211733 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:08.215436 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:08.215447 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:08.215466 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:08.215985 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:08.216380 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:08.216386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:08.216390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:08.216625 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:08.217243 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:08.217254 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:08.217614 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 05:06:08.217899 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 05:06:08.218506 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":393,"threads_started":1}
I20260927 05:06:08.221258 3934 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36893
I20260927 05:06:08.221583 3934 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:08.222380 3934 heartbeater.cc:511] Master 127.0.97.62:36893 requested a full tablet report, sending...
I20260927 05:06:08.223872 3477 ts_manager.cc:194] Registered new tserver with Master: 297c6660df9a49ada87d711cd36c478b (127.0.97.3:32853)
I20260927 05:06:08.224735 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57811
I20260927 05:06:08.226945 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 05:06:08 dist-test-slave-hgs0 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485568, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 05:06:08 dist-test-slave-hgs0 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485568, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:08.278915 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:08.264620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.0.97.62:36893)
0927 05:06:08.264830 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:08.264835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:08.265286 (+ 451us) server_negotiation.cc:407] Connection header received
0927 05:06:08.265438 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:08.265442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:08.265508 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:08.265603 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:08.267017 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.267930 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.269342 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.269906 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.270029 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:08.273949 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:08.273955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:08.273959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:08.273997 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:08.276512 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:08.277233 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:08.277238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:08.277240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:08.277366 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:08.277753 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:08.277756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:08.277758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:08.278064 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:08.278271 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:08.278480 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 05:06:08.278684 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260927 05:06:08.279490 3477 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34200
I20260927 05:06:08.281073 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3538
Sep 27 05:06:08 dist-test-slave-hgs0 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485568, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:06:08.303081 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:08.281990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.0.97.1:42397)
0927 05:06:08.282616 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:08.282623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:08.282649 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:06:08.282703 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:08.282709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:08.282926 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:08.283052 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:08.284226 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.285235 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.286418 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.286999 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.287100 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:08.297060 (+ 9960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:08.297075 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:08.297092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:08.297141 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:08.300165 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:08.301038 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:08.301046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:08.301048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:08.301183 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:08.301639 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:08.301648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:08.301650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:08.302009 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:08.302243 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:08.302522 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 05:06:08.302766 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":314,"threads_started":1}
W20260927 05:06:08.304571 3661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49538
W20260927 05:06:08.305857 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:06:08.306061 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 3538
I20260927 05:06:08.322363 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3672
Sep 27 05:06:08 dist-test-slave-hgs0 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485568, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:06:08.340902 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:08.323305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.0.97.2:46161)
0927 05:06:08.323848 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:08.323855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:08.323882 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:06:08.323996 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:08.324005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:08.324248 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:08.324453 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:08.325852 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.326997 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.328675 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.329378 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.329548 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:08.334702 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:08.334718 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:08.334736 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:08.334789 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:08.337999 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:08.338771 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:08.338781 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:08.338784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:08.338945 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:08.339288 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:08.339291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:08.339294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:08.339707 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:08.340072 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:08.340280 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 05:06:08.340525 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":265,"threads_started":1}
W20260927 05:06:08.342316 3795 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48804
W20260927 05:06:08.343642 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3672: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:06:08.343842 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 3672
I20260927 05:06:08.360162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3806
Sep 27 05:06:08 dist-test-slave-hgs0 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485568, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:06:08.377723 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:08.361244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.0.97.3:32853)
0927 05:06:08.361839 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:08.361846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:08.361874 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:06:08.361933 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:08.361940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:08.362171 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:08.362353 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:08.363728 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.364848 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.366120 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:08.366846 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:08.366988 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:08.371486 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:08.371499 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:08.371516 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:08.371564 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:08.374590 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:08.375389 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:08.375398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:08.375401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:08.375541 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:08.375929 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:08.375994 (+ 65us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:08.376002 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:08.376487 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:08.376784 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:08.377124 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 05:06:08.377368 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":271,"threads_started":1}
W20260927 05:06:08.379204 3929 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33104
W20260927 05:06:08.380690 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3806: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:06:08.380916 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 3806
I20260927 05:06:08.397719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3445
I20260927 05:06:08.398978 3507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:08.515179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 3445
2026-09-27T05:06:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3912 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:06:08 dist-test-slave-hgs0 krb5kdc[3952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:06:08 dist-test-slave-hgs0 krb5kdc[3952](info): set up 2 sockets
Sep 27 05:06:08 dist-test-slave-hgs0 krb5kdc[3952](info): commencing operation
krb5kdc: starting...
W20260927 05:06:09.720502 388 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:06:09 dist-test-slave-hgs0 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:06:09.931388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35479
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:10.278158 3965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:10.278510 3965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:10.278678 3965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:10.286365 3965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:10.286494 3965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:10.286592 3965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:10.286686 3965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:10.286770 3965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:10.297451 3965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:10.298926 3965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:10.300534 3965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:10.313706 3970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:10.313817 3973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:10.313723 3971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:10.314457 3965 server_base.cc:1034] running on GCE node
I20260927 05:06:10.315268 3965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:10.316753 3965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:10.317992 3965 hybrid_clock.cc:648] HybridClock initialized: now 1790485570317918 us; error 61 us; skew 500 ppm
I20260927 05:06:10.318408 3965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:10 dist-test-slave-hgs0 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:10.327405 3965 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:06:10.329286 3965 webserver.cc:460] Webserver started at http://127.0.97.62:43123/ using document root <none> and password file <none>
I20260927 05:06:10.330132 3965 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:10.330308 3965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:10.330689 3965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:10.334372 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "ba0a4b2948e24623b827b557df151ccd"
format_stamp: "Formatted at 2026-09-27 05:06:10 on dist-test-slave-hgs0"
server_key: "a9c2eb4ae12e65cea3116613be25cb85"
server_key_iv: "d7f5e88f4a0ad39091ad898f4712df93"
server_key_version: "encryptionkey@0"
I20260927 05:06:10.335409 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "ba0a4b2948e24623b827b557df151ccd"
format_stamp: "Formatted at 2026-09-27 05:06:10 on dist-test-slave-hgs0"
server_key: "a9c2eb4ae12e65cea3116613be25cb85"
server_key_iv: "d7f5e88f4a0ad39091ad898f4712df93"
server_key_version: "encryptionkey@0"
I20260927 05:06:10.341814 3965 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 05:06:10.346091 3980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:10.347770 3965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:06:10.348073 3965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "ba0a4b2948e24623b827b557df151ccd"
format_stamp: "Formatted at 2026-09-27 05:06:10 on dist-test-slave-hgs0"
server_key: "a9c2eb4ae12e65cea3116613be25cb85"
server_key_iv: "d7f5e88f4a0ad39091ad898f4712df93"
server_key_version: "encryptionkey@0"
I20260927 05:06:10.348320 3965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:10.366936 3965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:10.371819 3965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:10.372164 3965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:10.392613 3965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35479
I20260927 05:06:10.392647 4031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35479 every 8 connection(s)
I20260927 05:06:10.394160 3965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:10.398351 4032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:10.402779 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 3965
I20260927 05:06:10.403054 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:10.403911 388 external_mini_cluster.cc:1428] Setting key 83e8c160cb044fe4893b4c39940fe1af
I20260927 05:06:10.413631 4032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd: Bootstrap starting.
I20260927 05:06:10.418334 4032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:10.419979 4032 log.cc:824] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:06:10 dist-test-slave-hgs0 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:10.423741 4032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd: No bootstrap required, opened a new log
I20260927 05:06:10.430539 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:10.406191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.0.97.62:35479)
0927 05:06:10.407003 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:10.407017 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:10.407057 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:06:10.408805 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:10.408832 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:10.409306 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:10.409783 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:10.411738 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:10.413461 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:10.415433 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:10.416313 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:10.416465 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:10.422052 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:10.422061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:10.422079 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:10.422116 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:10.425497 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:10.426547 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:10.426554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:10.426561 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:10.426768 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:10.427212 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:10.427215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:10.427217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:10.427900 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:10.428306 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:10.428825 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 05:06:10.429332 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":381,"threads_started":1}
I20260927 05:06:10.433640 4032 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0a4b2948e24623b827b557df151ccd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35479 } }
I20260927 05:06:10.434191 4032 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:10.434298 4032 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0a4b2948e24623b827b557df151ccd, State: Initialized, Role: FOLLOWER
I20260927 05:06:10.434865 4032 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0a4b2948e24623b827b557df151ccd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35479 } }
I20260927 05:06:10.435096 4032 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:10.435228 4032 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:10.435429 4032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:10.438228 4032 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0a4b2948e24623b827b557df151ccd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35479 } }
I20260927 05:06:10.438869 4032 leader_election.cc:302] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba0a4b2948e24623b827b557df151ccd; no voters:
I20260927 05:06:10.439486 4032 leader_election.cc:288] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:10.439632 4037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:10.440424 4037 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [term 1 LEADER]: Becoming Leader. State: Replica: ba0a4b2948e24623b827b557df151ccd, State: Running, Role: LEADER
I20260927 05:06:10.440973 4037 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0a4b2948e24623b827b557df151ccd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35479 } }
I20260927 05:06:10.441988 4032 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:10.444342 4039 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba0a4b2948e24623b827b557df151ccd. Latest consensus state: current_term: 1 leader_uuid: "ba0a4b2948e24623b827b557df151ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0a4b2948e24623b827b557df151ccd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35479 } } }
I20260927 05:06:10.444439 4038 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba0a4b2948e24623b827b557df151ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0a4b2948e24623b827b557df151ccd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35479 } } }
I20260927 05:06:10.444855 4038 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:10.444855 4039 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:10.446866 4043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:10.452771 4043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:10.464082 4043 catalog_manager.cc:1347] Generated new cluster ID: 5fcefa795f6544eca30116d1bde7985a
I20260927 05:06:10.464249 4043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:10.477905 4043 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:10.479427 4043 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:10.488811 4043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba0a4b2948e24623b827b557df151ccd: Generated new TSK 0
I20260927 05:06:10.489893 4043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:06:10.556777 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35479
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:10.931464 4058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:10.931778 4058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:10.931874 4058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:10.939667 4058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:10.939806 4058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:10.939980 4058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:10.951081 4058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:10.952623 4058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:10.954047 4058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:10.969013 4063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:10.968945 4066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:10.969049 4064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:10.970207 4058 server_base.cc:1034] running on GCE node
I20260927 05:06:10.971002 4058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:10.972188 4058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:10.973443 4058 hybrid_clock.cc:648] HybridClock initialized: now 1790485570973387 us; error 51 us; skew 500 ppm
I20260927 05:06:10.973809 4058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:10 dist-test-slave-hgs0 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:10.977586 4058 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:06:10.979408 4058 webserver.cc:460] Webserver started at http://127.0.97.1:43499/ using document root <none> and password file <none>
I20260927 05:06:10.980315 4058 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:10.980468 4058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:10.980844 4058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:10.984287 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "195ffd6bcf234555b4df4205517b90aa"
format_stamp: "Formatted at 2026-09-27 05:06:10 on dist-test-slave-hgs0"
server_key: "98d54d999f26815b7ad9ce1eb9d833f2"
server_key_iv: "467b250015b8addb55a01544ed9b89e5"
server_key_version: "encryptionkey@0"
I20260927 05:06:10.985234 4058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "195ffd6bcf234555b4df4205517b90aa"
format_stamp: "Formatted at 2026-09-27 05:06:10 on dist-test-slave-hgs0"
server_key: "98d54d999f26815b7ad9ce1eb9d833f2"
server_key_iv: "467b250015b8addb55a01544ed9b89e5"
server_key_version: "encryptionkey@0"
I20260927 05:06:10.991848 4058 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 05:06:10.995769 4073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:10.997301 4058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:10.997496 4058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "195ffd6bcf234555b4df4205517b90aa"
format_stamp: "Formatted at 2026-09-27 05:06:10 on dist-test-slave-hgs0"
server_key: "98d54d999f26815b7ad9ce1eb9d833f2"
server_key_iv: "467b250015b8addb55a01544ed9b89e5"
server_key_version: "encryptionkey@0"
I20260927 05:06:10.997741 4058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:11.015959 4058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:11.021050 4058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:11.021384 4058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:11.022534 4058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:11.024240 4058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:11.024330 4058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:11.024518 4058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:11.024580 4058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:11.060863 4058 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46373
I20260927 05:06:11.060901 4185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46373 every 8 connection(s)
I20260927 05:06:11.062429 4058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:11.072686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 4058
I20260927 05:06:11.072919 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:06:11.073689 388 external_mini_cluster.cc:1428] Setting key b2ff67b3b50cab7150f3e43493f219d8
Sep 27 05:06:11 dist-test-slave-hgs0 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:11.085481 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:11.065410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48005 (local address 127.0.97.62:35479)
0927 05:06:11.065926 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:11.065933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:11.067209 (+ 1276us) server_negotiation.cc:407] Connection header received
0927 05:06:11.069281 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:11.069286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:11.069375 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:11.069494 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:11.071106 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:11.071849 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:11.074478 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:11.075044 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:11.075206 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:11.078963 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:11.078968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:11.078972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:11.079007 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:11.081880 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:11.082807 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:11.082813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:11.082816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:11.082947 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:11.083453 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:11.083457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:11.083460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:11.083758 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:11.083930 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:11.084804 (+ 874us) server_negotiation.cc:299] Negotiation successful
0927 05:06:11.085055 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":378,"thread_start_us":250,"threads_started":1}
I20260927 05:06:11.086594 4189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:11.066129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35479 (local address 127.0.97.1:48005)
0927 05:06:11.067022 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:11.067066 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:11.068980 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:11.069754 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:11.069763 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:11.070305 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:11.070925 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:11.070944 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:11.072062 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:11.072066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:11.074299 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:11.074318 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:11.075264 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:11.075269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:11.075481 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:11.076294 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:11.076320 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:11.078700 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:11.082086 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:11.082099 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:11.082129 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:11.082646 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:11.083074 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:11.083079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:11.083084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:11.083311 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:11.083905 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:11.083914 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:11.084374 (+ 460us) client_negotiation.cc:769] Sending connection context
0927 05:06:11.084683 (+ 309us) client_negotiation.cc:240] Negotiation successful
0927 05:06:11.085343 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":364,"threads_started":1}
I20260927 05:06:11.088522 4186 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35479
I20260927 05:06:11.088884 4186 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:11.089715 4186 heartbeater.cc:511] Master 127.0.97.62:35479 requested a full tablet report, sending...
I20260927 05:06:11.092311 3997 ts_manager.cc:194] Registered new tserver with Master: 195ffd6bcf234555b4df4205517b90aa (127.0.97.1:46373)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:06:11.094213 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:06:11.124411 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35479
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:11.483647 4192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:11.484032 4192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:11.484241 4192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:11.491767 4192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:11.491972 4192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:11.492161 4192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:11.502532 4192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:11.504107 4192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:11.505687 4192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:11.520629 4198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:11.520699 4200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:11.520780 4197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:11.521929 4192 server_base.cc:1034] running on GCE node
I20260927 05:06:11.522719 4192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:11.523834 4192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:11.525084 4192 hybrid_clock.cc:648] HybridClock initialized: now 1790485571525028 us; error 44 us; skew 500 ppm
I20260927 05:06:11.525458 4192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:11 dist-test-slave-hgs0 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:11.529022 4192 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:06:11.530745 4192 webserver.cc:460] Webserver started at http://127.0.97.2:43879/ using document root <none> and password file <none>
I20260927 05:06:11.531550 4192 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:11.531728 4192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:11.532131 4192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:11.535411 4192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "711a15ab6d684de5abca388b8debf10b"
format_stamp: "Formatted at 2026-09-27 05:06:11 on dist-test-slave-hgs0"
server_key: "68ce4af36ae3e2fbcc271fce91ecc61a"
server_key_iv: "ff16e5e0c1157296601686f5df9c46f9"
server_key_version: "encryptionkey@0"
I20260927 05:06:11.536367 4192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "711a15ab6d684de5abca388b8debf10b"
format_stamp: "Formatted at 2026-09-27 05:06:11 on dist-test-slave-hgs0"
server_key: "68ce4af36ae3e2fbcc271fce91ecc61a"
server_key_iv: "ff16e5e0c1157296601686f5df9c46f9"
server_key_version: "encryptionkey@0"
I20260927 05:06:11.542320 4192 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:06:11.546634 4207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:11.548452 4192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:06:11.548674 4192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "711a15ab6d684de5abca388b8debf10b"
format_stamp: "Formatted at 2026-09-27 05:06:11 on dist-test-slave-hgs0"
server_key: "68ce4af36ae3e2fbcc271fce91ecc61a"
server_key_iv: "ff16e5e0c1157296601686f5df9c46f9"
server_key_version: "encryptionkey@0"
I20260927 05:06:11.548939 4192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:11.584142 4192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:11.589053 4192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:11.589402 4192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:11.590514 4192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:11.592140 4192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:11.592252 4192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:11.592428 4192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:11.592505 4192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:11.628736 4192 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38147
I20260927 05:06:11.628782 4319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38147 every 8 connection(s)
I20260927 05:06:11.630239 4192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:11.637329 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 4192
I20260927 05:06:11.637588 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:06:11.638070 388 external_mini_cluster.cc:1428] Setting key 42e460d940c9c8d1e60d35e4bbc6ec30
Sep 27 05:06:11 dist-test-slave-hgs0 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:11.654990 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:11.633438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35435 (local address 127.0.97.62:35479)
0927 05:06:11.633965 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:11.633970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:11.635159 (+ 1189us) server_negotiation.cc:407] Connection header received
0927 05:06:11.637478 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:11.637485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:11.637604 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:11.637735 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:11.639874 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:11.641036 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:11.642891 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:11.643745 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:11.643897 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:11.647901 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:11.647907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:11.647910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:11.648013 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:11.650728 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:11.651837 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:11.651845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:11.651849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:11.652055 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:11.652602 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:11.652606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:11.652608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:11.652980 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:11.653146 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:11.654064 (+ 918us) server_negotiation.cc:299] Negotiation successful
0927 05:06:11.654348 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":296,"threads_started":1}
I20260927 05:06:11.655805 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:11.634008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35479 (local address 127.0.97.2:35435)
0927 05:06:11.634948 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:11.634997 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:11.637139 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:11.638030 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:11.638040 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:11.638755 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:11.639665 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:11.639692 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:11.641236 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:11.641241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:11.642701 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:11.642715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:11.644027 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:11.644034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:11.644271 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:11.644984 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:11.645009 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:11.647591 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:11.650907 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:11.650921 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:11.650945 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:11.651661 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:11.652175 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:11.652184 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:11.652190 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:11.652475 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:11.653179 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:11.653188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:11.653613 (+ 425us) client_negotiation.cc:769] Sending connection context
0927 05:06:11.653954 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 05:06:11.654500 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":414,"threads_started":1}
I20260927 05:06:11.657702 4320 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35479
I20260927 05:06:11.658048 4320 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:06:11.658897 4320 heartbeater.cc:511] Master 127.0.97.62:35479 requested a full tablet report, sending...
I20260927 05:06:11.660650 3997 ts_manager.cc:194] Registered new tserver with Master: 711a15ab6d684de5abca388b8debf10b (127.0.97.2:38147)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:06:11.661706 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:06:11.688011 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35479
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:12.038067 4326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:12.038390 4326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:12.038489 4326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:12.046715 4326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:12.046861 4326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:12.046988 4326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:12.058245 4326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:12.059651 4326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:12.061290 4326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:12.075762 4331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:12.075726 4334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:12.075807 4332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:12.077332 4326 server_base.cc:1034] running on GCE node
I20260927 05:06:12.078054 4326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:12.079098 4326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:12.080297 4326 hybrid_clock.cc:648] HybridClock initialized: now 1790485572080247 us; error 39 us; skew 500 ppm
I20260927 05:06:12.080644 4326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:12 dist-test-slave-hgs0 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:12.083997 4326 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:06:12.085656 4326 webserver.cc:460] Webserver started at http://127.0.97.3:40625/ using document root <none> and password file <none>
I20260927 05:06:12.086402 4326 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:12.086514 4326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:12.086800 4326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:12.090188 4326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "8288e7e4c877443fa48f607f0b60a403"
format_stamp: "Formatted at 2026-09-27 05:06:12 on dist-test-slave-hgs0"
server_key: "efccdacbb5ea56497206b454f70cc5f3"
server_key_iv: "b7d6c61c0c5a4c014569877c03966514"
server_key_version: "encryptionkey@0"
I20260927 05:06:12.091084 4326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8288e7e4c877443fa48f607f0b60a403"
format_stamp: "Formatted at 2026-09-27 05:06:12 on dist-test-slave-hgs0"
server_key: "efccdacbb5ea56497206b454f70cc5f3"
server_key_iv: "b7d6c61c0c5a4c014569877c03966514"
server_key_version: "encryptionkey@0"
I20260927 05:06:12.097296 4326 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 05:06:12.097658 4186 heartbeater.cc:503] Master 127.0.97.62:35479 was elected leader, sending a full tablet report...
I20260927 05:06:12.101732 4341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:12.103425 4326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260927 05:06:12.103627 4326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "8288e7e4c877443fa48f607f0b60a403"
format_stamp: "Formatted at 2026-09-27 05:06:12 on dist-test-slave-hgs0"
server_key: "efccdacbb5ea56497206b454f70cc5f3"
server_key_iv: "b7d6c61c0c5a4c014569877c03966514"
server_key_version: "encryptionkey@0"
I20260927 05:06:12.103865 4326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:12.120741 4326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:12.125723 4326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:12.126040 4326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:12.127080 4326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:12.128600 4326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:12.128708 4326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:12.128854 4326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:12.128924 4326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:12.164786 4326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40183
I20260927 05:06:12.164880 4453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40183 every 8 connection(s)
I20260927 05:06:12.166236 4326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:12.170588 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 4326
I20260927 05:06:12.170797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:06:12.171340 388 external_mini_cluster.cc:1428] Setting key c5e6f0e19fc07c63582c9e7edd26efd9
Sep 27 05:06:12 dist-test-slave-hgs0 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:12.189288 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:12.169109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46653 (local address 127.0.97.62:35479)
0927 05:06:12.169620 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:12.169626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:12.170947 (+ 1321us) server_negotiation.cc:407] Connection header received
0927 05:06:12.173261 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:12.173265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:12.173339 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:12.173462 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:12.175378 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.176272 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.178008 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.178432 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.178538 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:12.182660 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:12.182666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:12.182670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:12.182707 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:12.185740 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.186724 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.186730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.186734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.186901 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.187462 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.187468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.187471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.187828 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:12.188011 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:12.188797 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 05:06:12.188987 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":238,"threads_started":1}
I20260927 05:06:12.190388 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:12.169787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35479 (local address 127.0.97.3:46653)
0927 05:06:12.170750 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:12.170811 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:12.172910 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:12.173796 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:12.173812 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:12.174400 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:12.175189 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:12.175209 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.176433 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.176454 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:12.177863 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:12.177876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.178694 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.178700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:12.178944 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:12.179779 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:12.179805 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:12.182400 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:12.185908 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:12.185920 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:12.185940 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:12.186571 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:12.187031 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:12.187037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:12.187040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:12.187303 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:12.187995 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:12.188005 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:12.188403 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 05:06:12.188708 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 05:06:12.189141 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":358,"threads_started":1}
I20260927 05:06:12.192030 4454 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35479
I20260927 05:06:12.192404 4454 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:12.193269 4454 heartbeater.cc:511] Master 127.0.97.62:35479 requested a full tablet report, sending...
I20260927 05:06:12.195026 3997 ts_manager.cc:194] Registered new tserver with Master: 8288e7e4c877443fa48f607f0b60a403 (127.0.97.3:40183)
I20260927 05:06:12.196023 3997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46653
I20260927 05:06:12.200367 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 05:06:12 dist-test-slave-hgs0 krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 05:06:12 dist-test-slave-hgs0 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:06:12.252811 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:12.236403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.0.97.1:46373)
0927 05:06:12.237157 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:12.237169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:12.237203 (+ 34us) server_negotiation.cc:407] Connection header received
0927 05:06:12.237267 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:12.237274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:12.237517 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:12.237703 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:12.239239 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.240402 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.241721 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.242436 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.242584 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:12.246702 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:12.246717 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:12.246733 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:12.246780 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:12.249773 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.250525 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.250533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.250536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.250663 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.251077 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.251083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.251086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.251451 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:12.251777 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:12.252165 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 05:06:12.252449 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":359,"threads_started":1}
Sep 27 05:06:12 dist-test-slave-hgs0 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:12.273510 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:12.257554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.0.97.62:35479)
0927 05:06:12.257782 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:12.257788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:12.258115 (+ 327us) server_negotiation.cc:407] Connection header received
0927 05:06:12.258363 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:12.258369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:12.258444 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:12.258540 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:12.260622 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.261477 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.262868 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.263358 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.263495 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:12.268051 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:12.268057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:12.268061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:12.268100 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:12.271045 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.271742 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.271746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.271748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.271864 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.272232 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.272235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.272237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.272549 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:12.272725 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:12.272984 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 05:06:12.273155 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 05:06:12.274099 3997 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34736
I20260927 05:06:12.275225 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 4058
I20260927 05:06:12.289283 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:12.276110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.97.1:46373)
0927 05:06:12.276279 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:12.276284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:12.276609 (+ 325us) server_negotiation.cc:407] Connection header received
0927 05:06:12.276753 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:12.276757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:12.276820 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:12.276903 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:12.278143 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.278906 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.280196 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.280628 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.280735 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:12.283181 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:12.283187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:12.283191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:12.283241 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:12.286244 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.287026 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.287031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.287034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.287199 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.287690 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.287697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.287700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.288298 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:12.288525 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:12.288811 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 05:06:12.289016 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:06:12.289915 4181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:12.375252 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 4058
I20260927 05:06:12.392818 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 4192
Sep 27 05:06:12 dist-test-slave-hgs0 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:06:12.411777 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:12.393940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.0.97.2:38147)
0927 05:06:12.394579 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:12.394589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:12.394622 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:06:12.394704 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:12.394713 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:12.394990 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:12.395193 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:12.396540 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.398192 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.399595 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.400629 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.400783 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:12.405482 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:12.405496 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:12.405513 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:12.405555 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:12.408784 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.409544 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.409552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.409555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.409683 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.410081 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.410087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.410091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.410461 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:12.410780 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:12.411116 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 05:06:12.411412 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":313,"threads_started":1}
I20260927 05:06:12.413131 4315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:12.500953 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 4192
I20260927 05:06:12.518224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 4326
Sep 27 05:06:12 dist-test-slave-hgs0 krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:06:12.536967 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:12.519239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.0.97.3:40183)
0927 05:06:12.520028 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:12.520036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:12.520069 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:06:12.520134 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:12.520141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:12.520398 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:12.520548 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:12.521941 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.523073 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.524759 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:12.525440 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:12.525564 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:12.530657 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:12.530674 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:12.530692 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:12.530736 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:12.534046 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.534814 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.534822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.534824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.534947 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:12.535346 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:12.535349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:12.535351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:12.535727 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:12.536047 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:12.536426 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 05:06:12.536671 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":407,"threads_started":1}
I20260927 05:06:12.538335 4449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:12.626230 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 4326
I20260927 05:06:12.643790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 3965
I20260927 05:06:12.645114 4027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:12.755110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 3965
2026-09-27T05:06:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4233 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:06:12 dist-test-slave-hgs0 krb5kdc[4474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:06:12 dist-test-slave-hgs0 krb5kdc[4474](info): set up 2 sockets
Sep 27 05:06:12 dist-test-slave-hgs0 krb5kdc[4474](info): commencing operation
krb5kdc: starting...
W20260927 05:06:13.884588 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:06:14 dist-test-slave-hgs0 krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:06:14.100018 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34123
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:14.461503 4487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:14.461894 4487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:14.462075 4487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:14.469470 4487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:14.469657 4487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:14.469779 4487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:14.469885 4487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:14.469957 4487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:14.480216 4487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:14.481693 4487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:14.483132 4487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:14.497028 4492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:14.497028 4495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:14.497105 4493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:14.498410 4487 server_base.cc:1034] running on GCE node
I20260927 05:06:14.499321 4487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:14.501175 4487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:14.502439 4487 hybrid_clock.cc:648] HybridClock initialized: now 1790485574502380 us; error 50 us; skew 500 ppm
I20260927 05:06:14.502846 4487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:14 dist-test-slave-hgs0 krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:14.506443 4487 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:06:14.507704 4487 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:14.507812 4487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:14.508149 4487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:14.511166 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "9f7907a102e2467abfa3052f5534e6a6"
format_stamp: "Formatted at 2026-09-27 05:06:14 on dist-test-slave-hgs0"
server_key: "229fc2986229bd425ed6844076817860"
server_key_iv: "64b2578beb1c5346cc60acd8afbbb60b"
server_key_version: "encryptionkey@0"
I20260927 05:06:14.511920 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "9f7907a102e2467abfa3052f5534e6a6"
format_stamp: "Formatted at 2026-09-27 05:06:14 on dist-test-slave-hgs0"
server_key: "229fc2986229bd425ed6844076817860"
server_key_iv: "64b2578beb1c5346cc60acd8afbbb60b"
server_key_version: "encryptionkey@0"
I20260927 05:06:14.517261 4487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:06:14.520968 4501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:14.522518 4487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:06:14.522686 4487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "9f7907a102e2467abfa3052f5534e6a6"
format_stamp: "Formatted at 2026-09-27 05:06:14 on dist-test-slave-hgs0"
server_key: "229fc2986229bd425ed6844076817860"
server_key_iv: "64b2578beb1c5346cc60acd8afbbb60b"
server_key_version: "encryptionkey@0"
I20260927 05:06:14.522895 4487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:14.573133 4487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:14.577327 4487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:14.577553 4487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:14.596740 4487 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34123
I20260927 05:06:14.596781 4552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34123 every 8 connection(s)
I20260927 05:06:14.597843 4487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:14.602483 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:14.602809 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 4487
I20260927 05:06:14.603066 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:14.603996 388 external_mini_cluster.cc:1428] Setting key 08b5e8b24803976874fcae6a5cab524a
I20260927 05:06:14.617727 4553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6: Bootstrap starting.
Sep 27 05:06:14 dist-test-slave-hgs0 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:14.621356 4553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:14.622648 4553 log.cc:824] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:14.626448 4553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6: No bootstrap required, opened a new log
I20260927 05:06:14.631119 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:14.606706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.0.97.62:34123)
0927 05:06:14.607754 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:14.607770 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:14.607850 (+ 80us) server_negotiation.cc:407] Connection header received
0927 05:06:14.610441 (+ 2591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:14.610473 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:14.611044 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:14.611572 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:14.613295 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:14.614778 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:14.616279 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:14.617057 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:14.617292 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:14.621590 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:14.621603 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:14.621631 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:14.621677 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:14.625472 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:14.626375 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:14.626386 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:14.626397 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:14.626711 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:14.627219 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:14.627224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:14.627227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:14.628116 (+ 889us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:14.628460 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:14.628927 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 05:06:14.629479 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":491,"threads_started":1}
I20260927 05:06:14.636343 4553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7907a102e2467abfa3052f5534e6a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34123 } }
I20260927 05:06:14.636750 4553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:14.636873 4553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7907a102e2467abfa3052f5534e6a6, State: Initialized, Role: FOLLOWER
I20260927 05:06:14.637436 4553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7907a102e2467abfa3052f5534e6a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34123 } }
I20260927 05:06:14.637656 4553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:14.637759 4553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:14.637893 4553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:14.640650 4553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7907a102e2467abfa3052f5534e6a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34123 } }
I20260927 05:06:14.641156 4553 leader_election.cc:302] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f7907a102e2467abfa3052f5534e6a6; no voters:
I20260927 05:06:14.641741 4553 leader_election.cc:288] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:14.641911 4558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:14.642577 4558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [term 1 LEADER]: Becoming Leader. State: Replica: 9f7907a102e2467abfa3052f5534e6a6, State: Running, Role: LEADER
I20260927 05:06:14.643138 4558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7907a102e2467abfa3052f5534e6a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34123 } }
I20260927 05:06:14.644007 4553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:14.646086 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f7907a102e2467abfa3052f5534e6a6. Latest consensus state: current_term: 1 leader_uuid: "9f7907a102e2467abfa3052f5534e6a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7907a102e2467abfa3052f5534e6a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34123 } } }
I20260927 05:06:14.646449 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:14.647306 4559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f7907a102e2467abfa3052f5534e6a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7907a102e2467abfa3052f5534e6a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34123 } } }
I20260927 05:06:14.647703 4559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:14.649351 4564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:14.655723 4564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:14.667202 4564 catalog_manager.cc:1347] Generated new cluster ID: d9cc87b841d34854b27915b58d198f68
I20260927 05:06:14.667346 4564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:14.685794 4564 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:14.687496 4564 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:14.699995 4564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6: Generated new TSK 0
I20260927 05:06:14.701000 4564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:06:14.768397 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34123
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:15.132584 4579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:15.132902 4579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:15.133061 4579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:15.141455 4579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:15.141587 4579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:15.141785 4579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:15.152733 4579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:15.154235 4579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:15.155699 4579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:15.170210 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:15.170274 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:15.170181 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:15.173789 4579 server_base.cc:1034] running on GCE node
I20260927 05:06:15.174528 4579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:15.175647 4579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:15.176883 4579 hybrid_clock.cc:648] HybridClock initialized: now 1790485575176804 us; error 71 us; skew 500 ppm
I20260927 05:06:15.177259 4579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:15 dist-test-slave-hgs0 krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:15.180815 4579 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:06:15.182199 4579 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:15.182344 4579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:15.182684 4579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:15.185952 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "b1c3785f41424f6fb10f7704fe3e930a"
format_stamp: "Formatted at 2026-09-27 05:06:15 on dist-test-slave-hgs0"
server_key: "b2cf50096eb91f365fab262aa99d82b9"
server_key_iv: "3e09f7046f40030f13d8f1a2d22a13b5"
server_key_version: "encryptionkey@0"
I20260927 05:06:15.186853 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b1c3785f41424f6fb10f7704fe3e930a"
format_stamp: "Formatted at 2026-09-27 05:06:15 on dist-test-slave-hgs0"
server_key: "b2cf50096eb91f365fab262aa99d82b9"
server_key_iv: "3e09f7046f40030f13d8f1a2d22a13b5"
server_key_version: "encryptionkey@0"
I20260927 05:06:15.192658 4579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:06:15.196836 4593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:15.198652 4579 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:06:15.198882 4579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "b1c3785f41424f6fb10f7704fe3e930a"
format_stamp: "Formatted at 2026-09-27 05:06:15 on dist-test-slave-hgs0"
server_key: "b2cf50096eb91f365fab262aa99d82b9"
server_key_iv: "3e09f7046f40030f13d8f1a2d22a13b5"
server_key_version: "encryptionkey@0"
I20260927 05:06:15.199124 4579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:15.212520 4579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:15.217144 4579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:15.217420 4579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:15.218393 4579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:15.219802 4579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:15.219921 4579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:15.220069 4579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:15.220175 4579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:15.256048 4579 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46755
I20260927 05:06:15.256126 4705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46755 every 8 connection(s)
I20260927 05:06:15.256951 4579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:15.259799 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 4579
I20260927 05:06:15.260023 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:06:15.260825 388 external_mini_cluster.cc:1428] Setting key 98e57a234493351c75810c0083b7a893
Sep 27 05:06:15 dist-test-slave-hgs0 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:15.277897 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:15.259542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34113 (local address 127.0.97.62:34123)
0927 05:06:15.260095 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:15.260107 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:15.261366 (+ 1259us) server_negotiation.cc:407] Connection header received
0927 05:06:15.263474 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:15.263479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:15.263567 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:15.263682 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:15.265295 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:15.266012 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:15.267632 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:15.268083 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:15.268225 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:15.272044 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:15.272050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:15.272053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:15.272086 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:15.274630 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:15.275459 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:15.275471 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:15.275474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:15.275588 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:15.276098 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:15.276105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:15.276108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:15.276395 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:15.276548 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:15.277301 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 05:06:15.277533 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":272,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:06:15.279006 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:15.260355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34123 (local address 127.0.97.1:34113)
0927 05:06:15.261177 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:15.261236 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:15.263202 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:15.263981 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:15.264000 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:15.264531 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:15.265122 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:15.265140 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:15.266191 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:15.266195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:15.267460 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:15.267472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:15.268307 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:15.268317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:15.268559 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:15.269303 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:15.269326 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:15.271764 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:15.274820 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:15.274830 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:15.274850 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:15.275322 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:15.275718 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:15.275723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:15.275727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:15.275929 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:15.276535 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:15.276543 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:15.276924 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 05:06:15.277194 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 05:06:15.277758 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":374,"threads_started":1}
I20260927 05:06:15.280696 4706 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34123
I20260927 05:06:15.281019 4706 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:06:15.281817 4706 heartbeater.cc:511] Master 127.0.97.62:34123 requested a full tablet report, sending...
I20260927 05:06:15.284288 4518 ts_manager.cc:194] Registered new tserver with Master: b1c3785f41424f6fb10f7704fe3e930a (127.0.97.1:46755)
I20260927 05:06:15.286516 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:06:15.307996 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34123
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:15.667320 4712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:15.667709 4712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:15.667899 4712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:15.675509 4712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:15.675714 4712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:15.675923 4712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:15.686697 4712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:15.688270 4712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:15.689743 4712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:15.703622 4718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:15.703692 4717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:15.703719 4720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:15.704649 4712 server_base.cc:1034] running on GCE node
I20260927 05:06:15.705338 4712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:15.706342 4712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:15.707582 4712 hybrid_clock.cc:648] HybridClock initialized: now 1790485575707531 us; error 45 us; skew 500 ppm
I20260927 05:06:15.707929 4712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:15 dist-test-slave-hgs0 krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:15.715220 4712 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:06:15.716614 4712 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:15.716717 4712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:15.716995 4712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:15.720185 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "9e0091caf1934df89eafc08eb4ce1321"
format_stamp: "Formatted at 2026-09-27 05:06:15 on dist-test-slave-hgs0"
server_key: "4a8f97390ed0483ee7661acd4bcd8f0e"
server_key_iv: "590eadcf110358b5c7365ee648e6a565"
server_key_version: "encryptionkey@0"
I20260927 05:06:15.721025 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9e0091caf1934df89eafc08eb4ce1321"
format_stamp: "Formatted at 2026-09-27 05:06:15 on dist-test-slave-hgs0"
server_key: "4a8f97390ed0483ee7661acd4bcd8f0e"
server_key_iv: "590eadcf110358b5c7365ee648e6a565"
server_key_version: "encryptionkey@0"
I20260927 05:06:15.726815 4712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:06:15.731148 4726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:15.732909 4712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:06:15.733072 4712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "9e0091caf1934df89eafc08eb4ce1321"
format_stamp: "Formatted at 2026-09-27 05:06:15 on dist-test-slave-hgs0"
server_key: "4a8f97390ed0483ee7661acd4bcd8f0e"
server_key_iv: "590eadcf110358b5c7365ee648e6a565"
server_key_version: "encryptionkey@0"
I20260927 05:06:15.733331 4712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:15.747479 4712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:15.752218 4712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:15.752471 4712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:15.753427 4712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:15.754840 4712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:15.754930 4712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:15.755003 4712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:15.755059 4712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:15.789566 4712 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39209
I20260927 05:06:15.789587 4838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39209 every 8 connection(s)
I20260927 05:06:15.790491 4712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:15.799817 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 4712
I20260927 05:06:15.800052 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:06:15.800575 388 external_mini_cluster.cc:1428] Setting key 60a5bd1324fa6214cd4c30e761e7a524
Sep 27 05:06:15 dist-test-slave-hgs0 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:15.813253 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:15.793413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36679 (local address 127.0.97.62:34123)
0927 05:06:15.793896 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:15.793902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:15.795044 (+ 1142us) server_negotiation.cc:407] Connection header received
0927 05:06:15.797188 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:15.797194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:15.797322 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:15.797453 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:15.799174 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:15.800217 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:15.802147 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:15.802755 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:15.802867 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:15.807371 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:15.807377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:15.807381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:15.807414 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:15.810080 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:15.810858 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:15.810865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:15.810874 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:15.810992 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:15.811469 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:15.811472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:15.811475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:15.811745 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:15.811900 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:15.812652 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 05:06:15.812885 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":237,"threads_started":1}
I20260927 05:06:15.814330 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:15.794048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34123 (local address 127.0.97.2:36679)
0927 05:06:15.794849 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:15.794899 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:15.796866 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:15.797727 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:15.797737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:15.798329 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:15.798971 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:15.799002 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:15.800371 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:15.800375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:15.801984 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:15.801999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:15.803006 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:15.803014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:15.803365 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:15.804244 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:15.804294 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:15.807079 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:15.810249 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:15.810259 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:15.810278 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:15.810719 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:15.811108 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:15.811113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:15.811117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:15.811309 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:15.811876 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:15.811884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:15.812290 (+ 406us) client_negotiation.cc:769] Sending connection context
0927 05:06:15.812577 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 05:06:15.813118 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":320,"threads_started":1}
I20260927 05:06:15.816224 4839 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34123
I20260927 05:06:15.816552 4839 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:15.817272 4839 heartbeater.cc:511] Master 127.0.97.62:34123 requested a full tablet report, sending...
I20260927 05:06:15.818908 4518 ts_manager.cc:194] Registered new tserver with Master: 9e0091caf1934df89eafc08eb4ce1321 (127.0.97.2:39209)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:06:15.820135 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:06:15.848271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34123
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:16.209771 4845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:16.210125 4845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:16.210235 4845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:16.218222 4845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:16.218357 4845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:16.218483 4845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:16.229151 4845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:16.230558 4845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:16.231992 4845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:16.248256 4853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:16.248256 4851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:16.248627 4850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:16.249321 4845 server_base.cc:1034] running on GCE node
I20260927 05:06:16.250020 4845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:16.251259 4845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:16.252490 4845 hybrid_clock.cc:648] HybridClock initialized: now 1790485576252406 us; error 73 us; skew 500 ppm
I20260927 05:06:16.252883 4845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:16 dist-test-slave-hgs0 krb5kdc[4474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:16.256778 4845 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:06:16.258353 4845 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:16.258491 4845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:16.258885 4845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:16.263353 4845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "c01958608c914e819a7c279c8e5d3fa1"
format_stamp: "Formatted at 2026-09-27 05:06:16 on dist-test-slave-hgs0"
server_key: "af5dbc0f8f880d2d98c80f5bd7b68797"
server_key_iv: "f7ad3638b70e3a20d7d37c701cc81b86"
server_key_version: "encryptionkey@0"
I20260927 05:06:16.264418 4845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c01958608c914e819a7c279c8e5d3fa1"
format_stamp: "Formatted at 2026-09-27 05:06:16 on dist-test-slave-hgs0"
server_key: "af5dbc0f8f880d2d98c80f5bd7b68797"
server_key_iv: "f7ad3638b70e3a20d7d37c701cc81b86"
server_key_version: "encryptionkey@0"
I20260927 05:06:16.270644 4845 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:06:16.274588 4859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:16.276208 4845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:06:16.276427 4845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "c01958608c914e819a7c279c8e5d3fa1"
format_stamp: "Formatted at 2026-09-27 05:06:16 on dist-test-slave-hgs0"
server_key: "af5dbc0f8f880d2d98c80f5bd7b68797"
server_key_iv: "f7ad3638b70e3a20d7d37c701cc81b86"
server_key_version: "encryptionkey@0"
I20260927 05:06:16.276680 4845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:16.289878 4706 heartbeater.cc:503] Master 127.0.97.62:34123 was elected leader, sending a full tablet report...
I20260927 05:06:16.300807 4845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:16.306457 4845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:16.306788 4845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:16.308058 4845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:16.309566 4845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:16.309701 4845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:16.309840 4845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:16.309926 4845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:16.345566 4845 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41651
I20260927 05:06:16.345615 4971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41651 every 8 connection(s)
I20260927 05:06:16.346479 4845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:16.350699 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 4845
I20260927 05:06:16.350919 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:06:16.351353 388 external_mini_cluster.cc:1428] Setting key 85779625a5a22707b2e22571fd9cadbd
Sep 27 05:06:16 dist-test-slave-hgs0 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:16.368209 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.349057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50191 (local address 127.0.97.62:34123)
0927 05:06:16.349527 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:16.349532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:16.350723 (+ 1191us) server_negotiation.cc:407] Connection header received
0927 05:06:16.352977 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:16.352982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:16.353088 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:16.353224 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:16.354862 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.355615 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.357343 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.357793 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.357935 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.361873 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:16.361878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:16.361882 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:16.361919 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:16.364681 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:16.365599 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:16.365604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:16.365606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:16.365751 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:16.366373 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:16.366378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:16.366382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:16.366720 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:16.366899 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:16.367669 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 05:06:16.367868 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":223,"threads_started":1}
I20260927 05:06:16.369278 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.349702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34123 (local address 127.0.97.3:50191)
0927 05:06:16.350541 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:16.350586 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:16.352695 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:16.353502 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:16.353511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:16.354050 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:16.354669 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.354687 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.355785 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.355792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.357163 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.357176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.358048 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.358055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.358303 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.359019 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:16.359043 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:16.361581 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:16.364882 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:16.364893 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:16.364920 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:16.365436 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:16.365881 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:16.365887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:16.365891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:16.366190 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:16.366884 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:16.366899 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:16.367285 (+ 386us) client_negotiation.cc:769] Sending connection context
0927 05:06:16.367550 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 05:06:16.368072 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":334,"threads_started":1}
I20260927 05:06:16.370787 4972 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34123
I20260927 05:06:16.371120 4972 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:16.371806 4972 heartbeater.cc:511] Master 127.0.97.62:34123 requested a full tablet report, sending...
I20260927 05:06:16.373308 4518 ts_manager.cc:194] Registered new tserver with Master: c01958608c914e819a7c279c8e5d3fa1 (127.0.97.3:41651)
I20260927 05:06:16.374135 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50191
I20260927 05:06:16.379242 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:06:16.396054 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.383829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.0.97.62:34123)
0927 05:06:16.384052 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:16.384057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:16.384544 (+ 487us) server_negotiation.cc:407] Connection header received
0927 05:06:16.384693 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:16.384697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:16.384806 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:16.384907 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:16.386404 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.387187 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.388505 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.388947 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.389063 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.391261 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:16.391266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:16.391269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:16.391300 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:16.393795 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:16.394408 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:16.394411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:16.394414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:16.394521 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:16.394850 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:16.394852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:16.394854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:16.395153 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:16.395352 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:16.395580 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 05:06:16.395738 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:06:16.400381 4518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:06:16.407609 4518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:06:16.442029 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.422915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.0.97.2:39209)
0927 05:06:16.432593 (+ 9678us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:16.432603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:16.433389 (+ 786us) server_negotiation.cc:407] Connection header received
0927 05:06:16.433499 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:16.433508 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:16.433786 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:16.434002 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:16.434982 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.436512 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.439492 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.440851 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.441010 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.441221 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:16.441503 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 05:06:16.441670 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9502,"thread_start_us":9363,"threads_started":1}
I20260927 05:06:16.442070 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.422316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39209 (local address 127.0.0.1:41000)
0927 05:06:16.433082 (+ 10766us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:16.433107 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:16.433239 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:16.434080 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:16.434087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:16.434121 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:16.434789 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.434805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.436687 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.436696 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.439331 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.439343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.441042 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.441048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.441253 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.441306 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 05:06:16.441459 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 05:06:16.441663 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10622,"mutex_wait_us":300,"thread_start_us":10467,"threads_started":1}
I20260927 05:06:16.443568 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.421537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41651 (local address 127.0.0.1:50308)
0927 05:06:16.431573 (+ 10036us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:16.431601 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:16.431749 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:16.435407 (+ 3658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:16.435411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:16.435438 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:16.435843 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.435853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.437853 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.437861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.440337 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.440351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.442721 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.442728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.442904 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.442933 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 05:06:16.443017 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 05:06:16.443142 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9868,"mutex_wait_us":52,"thread_start_us":9581,"threads_started":1}
I20260927 05:06:16.443594 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.420758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.0.97.1:46755)
0927 05:06:16.432053 (+ 11295us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:16.432065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:16.432098 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:06:16.432226 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:16.432237 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:16.433246 (+ 1009us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:16.434546 (+ 1300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:16.434779 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.436459 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.439999 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.441621 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.441778 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.442018 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:16.443009 (+ 991us) server_negotiation.cc:299] Negotiation successful
0927 05:06:16.443232 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11050,"thread_start_us":11078,"threads_started":1}
I20260927 05:06:16.443162 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.420535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46755 (local address 127.0.0.1:57370)
0927 05:06:16.428461 (+ 7926us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:16.428533 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:16.428854 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:16.433622 (+ 4768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:16.433635 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:16.433684 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:16.434218 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.434251 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.436615 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.436621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.439736 (+ 3115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.439779 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.442192 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.442200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.442367 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.442400 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:06:16.442500 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 05:06:16.442661 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":105,"thread_start_us":340,"threads_started":1}
I20260927 05:06:16.444082 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.422081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.0.97.3:41651)
0927 05:06:16.434837 (+ 12756us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:16.434847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:16.434882 (+ 35us) server_negotiation.cc:407] Connection header received
0927 05:06:16.435006 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:16.435012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:16.435209 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:16.435423 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:16.436069 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.437667 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.440533 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.442125 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.442640 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.442884 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:16.443287 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 05:06:16.443488 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12544,"thread_start_us":12374,"threads_started":1}
I20260927 05:06:16.447641 4641 tablet_service.cc:1467] Processing CreateTablet for tablet ea3614a6ffea40d1909f9ff001d0189d (DEFAULT_TABLE table=test-table [id=88180e30fc464482bf54b66ad33b9617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:16.447669 4907 tablet_service.cc:1467] Processing CreateTablet for tablet ea3614a6ffea40d1909f9ff001d0189d (DEFAULT_TABLE table=test-table [id=88180e30fc464482bf54b66ad33b9617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:16.447669 4774 tablet_service.cc:1467] Processing CreateTablet for tablet ea3614a6ffea40d1909f9ff001d0189d (DEFAULT_TABLE table=test-table [id=88180e30fc464482bf54b66ad33b9617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:16.449266 4907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3614a6ffea40d1909f9ff001d0189d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:16.449292 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3614a6ffea40d1909f9ff001d0189d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:16.449326 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea3614a6ffea40d1909f9ff001d0189d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:16.462239 4991 tablet_bootstrap.cc:492] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a: Bootstrap starting.
I20260927 05:06:16.465750 4992 tablet_bootstrap.cc:492] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: Bootstrap starting.
I20260927 05:06:16.468307 4991 tablet_bootstrap.cc:654] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:16.468788 4993 tablet_bootstrap.cc:492] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: Bootstrap starting.
I20260927 05:06:16.469995 4992 tablet_bootstrap.cc:654] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:16.470080 4991 log.cc:824] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:16.471369 4992 log.cc:824] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:16.472584 4993 tablet_bootstrap.cc:654] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:16.473722 4993 log.cc:824] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:16.473964 4991 tablet_bootstrap.cc:492] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a: No bootstrap required, opened a new log
I20260927 05:06:16.474311 4991 ts_tablet_manager.cc:1397] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260927 05:06:16.475049 4992 tablet_bootstrap.cc:492] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: No bootstrap required, opened a new log
I20260927 05:06:16.475358 4992 ts_tablet_manager.cc:1397] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 05:06:16.476980 4993 tablet_bootstrap.cc:492] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: No bootstrap required, opened a new log
I20260927 05:06:16.477275 4993 ts_tablet_manager.cc:1397] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260927 05:06:16.485509 4993 raft_consensus.cc:348] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c3785f41424f6fb10f7704fe3e930a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46755 } } peers { permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 } } peers { permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 } }
I20260927 05:06:16.485970 4993 raft_consensus.cc:374] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:16.486071 4993 raft_consensus.cc:729] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01958608c914e819a7c279c8e5d3fa1, State: Initialized, Role: FOLLOWER
I20260927 05:06:16.486799 4993 consensus_queue.cc:260] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c3785f41424f6fb10f7704fe3e930a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46755 } } peers { permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 } } peers { permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 } }
I20260927 05:06:16.487668 4991 raft_consensus.cc:348] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c3785f41424f6fb10f7704fe3e930a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46755 } } peers { permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 } } peers { permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 } }
I20260927 05:06:16.487668 4992 raft_consensus.cc:348] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c3785f41424f6fb10f7704fe3e930a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46755 } } peers { permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 } } peers { permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 } }
I20260927 05:06:16.488058 4972 heartbeater.cc:503] Master 127.0.97.62:34123 was elected leader, sending a full tablet report...
I20260927 05:06:16.488209 4991 raft_consensus.cc:374] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:16.488221 4992 raft_consensus.cc:374] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:16.488322 4991 raft_consensus.cc:729] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c3785f41424f6fb10f7704fe3e930a, State: Initialized, Role: FOLLOWER
I20260927 05:06:16.488322 4992 raft_consensus.cc:729] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e0091caf1934df89eafc08eb4ce1321, State: Initialized, Role: FOLLOWER
I20260927 05:06:16.488835 4993 ts_tablet_manager.cc:1428] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260927 05:06:16.489151 4991 consensus_queue.cc:260] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c3785f41424f6fb10f7704fe3e930a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46755 } } peers { permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 } } peers { permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 } }
I20260927 05:06:16.489153 4992 consensus_queue.cc:260] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c3785f41424f6fb10f7704fe3e930a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46755 } } peers { permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 } } peers { permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 } }
I20260927 05:06:16.491153 4839 heartbeater.cc:503] Master 127.0.97.62:34123 was elected leader, sending a full tablet report...
I20260927 05:06:16.492767 4992 ts_tablet_manager.cc:1428] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260927 05:06:16.495152 4991 ts_tablet_manager.cc:1428] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a: Time spent starting tablet: real 0.021s user 0.010s sys 0.007s
W20260927 05:06:16.509337 4707 tablet.cc:2369] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:06:16.542502 4840 tablet.cc:2369] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:06:16.598030 4973 tablet.cc:2369] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:06:16.725032 4998 raft_consensus.cc:482] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:16.725400 4998 raft_consensus.cc:504] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c3785f41424f6fb10f7704fe3e930a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46755 } } peers { permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 } } peers { permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 } }
I20260927 05:06:16.727350 4998 leader_election.cc:288] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c01958608c914e819a7c279c8e5d3fa1 (127.0.97.3:41651), 9e0091caf1934df89eafc08eb4ce1321 (127.0.97.2:39209)
I20260927 05:06:16.734598 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.727929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53887 (local address 127.0.97.2:39209)
0927 05:06:16.728181 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:16.728187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:16.728772 (+ 585us) server_negotiation.cc:407] Connection header received
0927 05:06:16.729020 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:16.729026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:16.729097 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:16.729218 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:16.729863 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.730647 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.732653 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.733511 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.733692 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.733804 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:16.734127 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 05:06:16.734288 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:06:16.734889 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.727850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54611 (local address 127.0.97.3:41651)
0927 05:06:16.728191 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:16.728200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:16.729206 (+ 1006us) server_negotiation.cc:407] Connection header received
0927 05:06:16.729419 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:16.729428 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:16.729488 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:16.729611 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:16.730212 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.730995 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.732958 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.733839 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.734007 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.734127 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:16.734451 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 05:06:16.734610 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260927 05:06:16.735025 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.727717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41651 (local address 127.0.97.1:54611)
0927 05:06:16.729089 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:16.729103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:16.729266 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:16.729635 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:16.729640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:16.729664 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:16.730034 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.730043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.731152 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.731172 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.732775 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.732787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.734085 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.734091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.734279 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.734305 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 05:06:16.734425 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 05:06:16.734576 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"mutex_wait_us":25,"thread_start_us":1071,"threads_started":1}
I20260927 05:06:16.735080 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.727695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39209 (local address 127.0.97.1:53887)
0927 05:06:16.728608 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:16.728625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:16.728859 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:16.729235 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:16.729239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:16.729335 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:16.729705 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.729715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.730796 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.730802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.732484 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:16.732496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.733764 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.733771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:16.733949 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.733975 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 05:06:16.734099 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 05:06:16.734261 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":3,"spinlock_wait_cycles":28928,"thread_start_us":617,"threads_started":1}
I20260927 05:06:16.736385 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3614a6ffea40d1909f9ff001d0189d" candidate_uuid: "b1c3785f41424f6fb10f7704fe3e930a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e0091caf1934df89eafc08eb4ce1321" is_pre_election: true
I20260927 05:06:16.736385 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3614a6ffea40d1909f9ff001d0189d" candidate_uuid: "b1c3785f41424f6fb10f7704fe3e930a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01958608c914e819a7c279c8e5d3fa1" is_pre_election: true
I20260927 05:06:16.737026 4927 raft_consensus.cc:2454] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1c3785f41424f6fb10f7704fe3e930a in term 0.
I20260927 05:06:16.737026 4794 raft_consensus.cc:2454] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1c3785f41424f6fb10f7704fe3e930a in term 0.
I20260927 05:06:16.737890 4597 leader_election.cc:302] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9e0091caf1934df89eafc08eb4ce1321, b1c3785f41424f6fb10f7704fe3e930a, c01958608c914e819a7c279c8e5d3fa1; no voters:
I20260927 05:06:16.738503 4998 raft_consensus.cc:2781] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:06:16.738701 4998 raft_consensus.cc:482] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:06:16.738847 4998 raft_consensus.cc:3037] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:16.741297 4998 raft_consensus.cc:504] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c3785f41424f6fb10f7704fe3e930a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46755 } } peers { permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 } } peers { permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 } }
I20260927 05:06:16.741993 4998 leader_election.cc:288] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [CANDIDATE]: Term 1 election: Requested vote from peers c01958608c914e819a7c279c8e5d3fa1 (127.0.97.3:41651), 9e0091caf1934df89eafc08eb4ce1321 (127.0.97.2:39209)
I20260927 05:06:16.742583 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3614a6ffea40d1909f9ff001d0189d" candidate_uuid: "b1c3785f41424f6fb10f7704fe3e930a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01958608c914e819a7c279c8e5d3fa1"
I20260927 05:06:16.742587 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea3614a6ffea40d1909f9ff001d0189d" candidate_uuid: "b1c3785f41424f6fb10f7704fe3e930a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e0091caf1934df89eafc08eb4ce1321"
I20260927 05:06:16.742774 4927 raft_consensus.cc:3037] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:16.742841 4794 raft_consensus.cc:3037] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:16.745252 4794 raft_consensus.cc:2454] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1c3785f41424f6fb10f7704fe3e930a in term 1.
I20260927 05:06:16.745365 4927 raft_consensus.cc:2454] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1c3785f41424f6fb10f7704fe3e930a in term 1.
I20260927 05:06:16.745849 4596 leader_election.cc:302] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e0091caf1934df89eafc08eb4ce1321, b1c3785f41424f6fb10f7704fe3e930a; no voters:
I20260927 05:06:16.746181 4998 raft_consensus.cc:2781] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:16.746639 4998 raft_consensus.cc:686] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 1 LEADER]: Becoming Leader. State: Replica: b1c3785f41424f6fb10f7704fe3e930a, State: Running, Role: LEADER
I20260927 05:06:16.747191 4998 consensus_queue.cc:237] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c3785f41424f6fb10f7704fe3e930a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46755 } } peers { permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 } } peers { permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 } }
I20260927 05:06:16.752364 4517 catalog_manager.cc:5495] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a reported cstate change: term changed from 0 to 1, leader changed from <none> to b1c3785f41424f6fb10f7704fe3e930a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b1c3785f41424f6fb10f7704fe3e930a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1c3785f41424f6fb10f7704fe3e930a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 } health_report { overall_health: UNKNOWN } } }
I20260927 05:06:16.808264 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.801796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.0.97.1:46755)
0927 05:06:16.802064 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:16.802071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:16.802192 (+ 121us) server_negotiation.cc:407] Connection header received
0927 05:06:16.802521 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:16.802526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:16.802627 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:16.802802 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:06:16.803329 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.804267 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.805896 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.806338 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.806447 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.806969 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:06:16.807600 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:06:16.807728 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:16.807886 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 05:06:16.808039 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260927 05:06:16.817288 4927 raft_consensus.cc:1261] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1 [term 1 FOLLOWER]: Refusing update from remote peer b1c3785f41424f6fb10f7704fe3e930a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:16.817288 4794 raft_consensus.cc:1261] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321 [term 1 FOLLOWER]: Refusing update from remote peer b1c3785f41424f6fb10f7704fe3e930a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:16.818437 5002 consensus_queue.cc:1035] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e0091caf1934df89eafc08eb4ce1321" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:16.818822 4998 consensus_queue.cc:1035] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c01958608c914e819a7c279c8e5d3fa1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:16.832167 5007 mvcc.cc:205] Tried to move back new op lower bound from 7333828922632310784 to 7333828922360799232. Current Snapshot: MvccSnapshot[applied={T|T < 7333828922632310784}]
I20260927 05:06:16.849684 4517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:16.849990 4517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:16.853741 4517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f7907a102e2467abfa3052f5534e6a6: Sending DeleteTablet for 3 replicas of tablet ea3614a6ffea40d1909f9ff001d0189d
I20260927 05:06:16.854935 4907 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3614a6ffea40d1909f9ff001d0189d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50308
I20260927 05:06:16.854959 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3614a6ffea40d1909f9ff001d0189d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57370
I20260927 05:06:16.855715 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet ea3614a6ffea40d1909f9ff001d0189d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41000
I20260927 05:06:16.855872 5014 tablet_replica.cc:330] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: stopping tablet replica
I20260927 05:06:16.856503 5014 raft_consensus.cc:2229] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:16.857120 5014 raft_consensus.cc:2258] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:16.858938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 4579
I20260927 05:06:16.865598 5014 ts_tablet_manager.cc:1905] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:16.871975 5016 tablet_replica.cc:330] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: stopping tablet replica
I20260927 05:06:16.872495 5016 raft_consensus.cc:2229] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:16.873041 5016 raft_consensus.cc:2258] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:16.879184 5014 ts_tablet_manager.cc:1918] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:16.879415 5014 log.cc:1196] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/ea3614a6ffea40d1909f9ff001d0189d
I20260927 05:06:16.882508 5016 ts_tablet_manager.cc:1905] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 05:06:16 dist-test-slave-hgs0 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:06:16.888553 5015 tablet_replica.cc:330] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a: stopping tablet replica
I20260927 05:06:16.889128 5015 raft_consensus.cc:2229] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:06:16.889988 5015 raft_consensus.cc:2258] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:16.890527 5014 ts_tablet_manager.cc:1939] T ea3614a6ffea40d1909f9ff001d0189d P c01958608c914e819a7c279c8e5d3fa1: Deleting consensus metadata
I20260927 05:06:16.893060 4505 catalog_manager.cc:4841] TS c01958608c914e819a7c279c8e5d3fa1 (127.0.97.3:41651): tablet ea3614a6ffea40d1909f9ff001d0189d (table test-table [id=88180e30fc464482bf54b66ad33b9617]) successfully deleted
I20260927 05:06:16.893469 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:16.860710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.0.97.1:46755)
0927 05:06:16.863372 (+ 2662us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:16.863379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:16.871648 (+ 8269us) server_negotiation.cc:407] Connection header received
0927 05:06:16.872264 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:16.872270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:16.872355 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:16.872442 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:16.874341 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.875513 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.877373 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:16.878092 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:16.878193 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:16.884047 (+ 5854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:16.884065 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:16.884084 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:16.884137 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:16.888587 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:16.889689 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:16.889700 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:16.889704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:16.889973 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:16.890584 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:16.890590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:16.890593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:16.891289 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:16.891637 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:16.892417 (+ 780us) server_negotiation.cc:299] Negotiation successful
0927 05:06:16.892963 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2508}
I20260927 05:06:16.894418 5015 ts_tablet_manager.cc:1905] T ea3614a6ffea40d1909f9ff001d0189d P b1c3785f41424f6fb10f7704fe3e930a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:16.894934 4701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:16.895849 5016 ts_tablet_manager.cc:1918] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:16.896198 5016 log.cc:1196] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/ea3614a6ffea40d1909f9ff001d0189d
I20260927 05:06:16.896895 5016 ts_tablet_manager.cc:1939] T ea3614a6ffea40d1909f9ff001d0189d P 9e0091caf1934df89eafc08eb4ce1321: Deleting consensus metadata
I20260927 05:06:16.899034 4504 catalog_manager.cc:4841] TS 9e0091caf1934df89eafc08eb4ce1321 (127.0.97.2:39209): tablet ea3614a6ffea40d1909f9ff001d0189d (table test-table [id=88180e30fc464482bf54b66ad33b9617]) successfully deleted
I20260927 05:06:17.001785 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 4579
I20260927 05:06:17.024847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 4712
W20260927 05:06:17.024986 4502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46755 (error 108)
I20260927 05:06:17.027793 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.026815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46755 (local address 127.0.0.1:57396)
0927 05:06:17.027479 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:17.027613 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":447,"threads_started":1}
W20260927 05:06:17.028133 4502 catalog_manager.cc:4584] TS b1c3785f41424f6fb10f7704fe3e930a (127.0.97.1:46755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea3614a6ffea40d1909f9ff001d0189d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46755: connect: Connection refused (error 111)
Sep 27 05:06:17 dist-test-slave-hgs0 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:06:17.041914 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.025890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.0.97.2:39209)
0927 05:06:17.026097 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:17.026102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:17.026202 (+ 100us) server_negotiation.cc:407] Connection header received
0927 05:06:17.026443 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:17.026453 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:17.026552 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:17.026719 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:17.028493 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:17.029277 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:17.030566 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:17.030959 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:17.031056 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:17.035644 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:17.035666 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:17.035688 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:17.035746 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:17.038837 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:17.039634 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:17.039646 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:17.039648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:17.039827 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:17.040348 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:17.040352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:17.040354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:17.040789 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:17.041018 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:17.041306 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 05:06:17.041478 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:06:17.042623 4834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:17.082051 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.081442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46755 (local address 127.0.0.1:57408)
0927 05:06:17.081774 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:17.081905 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 05:06:17.083256 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.082873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46755 (local address 127.0.0.1:57410)
0927 05:06:17.083037 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:17.083111 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 05:06:17.140244 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 4712
I20260927 05:06:17.144258 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.143571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46755 (local address 127.0.0.1:57422)
0927 05:06:17.143870 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:17.144041 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 05:06:17.145594 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.145053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46755 (local address 127.0.0.1:57432)
0927 05:06:17.145244 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:17.145353 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 05:06:17.158396 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 4845
Sep 27 05:06:17 dist-test-slave-hgs0 krb5kdc[4474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:06:17.175848 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.159171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.97.3:41651)
0927 05:06:17.159384 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:17.159391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:17.159469 (+ 78us) server_negotiation.cc:407] Connection header received
0927 05:06:17.159698 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:17.159703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:17.159780 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:17.159895 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:17.161210 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:17.162137 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:17.163542 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:17.164122 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:17.164290 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:17.169133 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:17.169152 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:17.169171 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:17.169216 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:17.172710 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:17.173565 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:17.173588 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:17.173590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:17.173749 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:17.174163 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:17.174169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:17.174173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:17.174578 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:17.174915 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:17.175251 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 05:06:17.175493 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 05:06:17.176651 4967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:17.226293 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.225667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46755 (local address 127.0.0.1:57438)
0927 05:06:17.225992 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:17.226106 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 05:06:17.227387 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.226931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46755 (local address 127.0.0.1:57446)
0927 05:06:17.227129 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:17.227219 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 05:06:17.273043 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 4845
I20260927 05:06:17.290916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 4487
I20260927 05:06:17.292353 4548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:17.407663 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.406741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46755 (local address 127.0.0.1:57460)
0927 05:06:17.407213 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:17.407393 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289}
I20260927 05:06:17.407868 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 4487
2026-09-27T05:06:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4654 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T05:06:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:17Z Disabled control of system clock
I20260927 05:06:17.480221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41455
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 05:06:17.843365 5027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:17.843704 5027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:17.843801 5027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:17.851310 5027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:17.851444 5027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:17.851488 5027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:17.851608 5027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:17.851715 5027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:17.861294 5027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:17.862807 5027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:17.864792 5027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:17.877897 5033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:17.877921 5035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:17.877843 5032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:17.882014 5027 server_base.cc:1034] running on GCE node
I20260927 05:06:17.882941 5027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:17.884599 5027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:17.885818 5027 hybrid_clock.cc:648] HybridClock initialized: now 1790485577885746 us; error 76 us; skew 500 ppm
I20260927 05:06:17.886219 5027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:06:17.888896 5027 webserver.cc:460] Webserver started at http://127.0.97.62:44455/ using document root <none> and password file <none>
I20260927 05:06:17.889665 5027 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:17.889788 5027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:17.890161 5027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:17.893447 5027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "7cdbc752525742088938798d3f513467"
format_stamp: "Formatted at 2026-09-27 05:06:17 on dist-test-slave-hgs0"
server_key: "6c6d907e0b05ccd1aa6eab8f230a59c1"
server_key_iv: "e1ff3cd5cd1d17e7efa3e19a77af4a79"
server_key_version: "encryptionkey@0"
I20260927 05:06:17.894332 5027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "7cdbc752525742088938798d3f513467"
format_stamp: "Formatted at 2026-09-27 05:06:17 on dist-test-slave-hgs0"
server_key: "6c6d907e0b05ccd1aa6eab8f230a59c1"
server_key_iv: "e1ff3cd5cd1d17e7efa3e19a77af4a79"
server_key_version: "encryptionkey@0"
I20260927 05:06:17.899896 5027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:06:17.903623 5041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:17.905249 5027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:17.905472 5027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "7cdbc752525742088938798d3f513467"
format_stamp: "Formatted at 2026-09-27 05:06:17 on dist-test-slave-hgs0"
server_key: "6c6d907e0b05ccd1aa6eab8f230a59c1"
server_key_iv: "e1ff3cd5cd1d17e7efa3e19a77af4a79"
server_key_version: "encryptionkey@0"
I20260927 05:06:17.905709 5027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:17.911720 5027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:17.912611 5027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:17.912921 5027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:17.932013 5027 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41455
I20260927 05:06:17.932044 5092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41455 every 8 connection(s)
I20260927 05:06:17.933530 5027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:17.937795 5093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:17.942274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 5027
I20260927 05:06:17.942485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:17.943140 388 external_mini_cluster.cc:1428] Setting key 4647ba54212fe6fb804481a5092073eb
I20260927 05:06:17.950752 5093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467: Bootstrap starting.
I20260927 05:06:17.954340 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:17.945376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.0.97.62:41455)
0927 05:06:17.946595 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:17.946616 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:17.946683 (+ 67us) server_negotiation.cc:407] Connection header received
0927 05:06:17.949141 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:17.949173 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:17.949764 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:17.950231 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:17.952528 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:17.952535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:17.952549 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:17.952577 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:17.952650 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:17.952685 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:17.952870 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:17.952987 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 05:06:17.953295 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":421,"threads_started":1}
I20260927 05:06:17.954756 5093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:17.955902 5093 log.cc:824] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:17.959108 5093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467: No bootstrap required, opened a new log
I20260927 05:06:17.966735 5093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbc752525742088938798d3f513467" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41455 } }
I20260927 05:06:17.967163 5093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:17.967289 5093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cdbc752525742088938798d3f513467, State: Initialized, Role: FOLLOWER
I20260927 05:06:17.967900 5093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbc752525742088938798d3f513467" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41455 } }
I20260927 05:06:17.968207 5093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:17.968366 5093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:17.968549 5093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:17.970705 5093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbc752525742088938798d3f513467" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41455 } }
I20260927 05:06:17.971235 5093 leader_election.cc:302] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cdbc752525742088938798d3f513467; no voters:
I20260927 05:06:17.971896 5093 leader_election.cc:288] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:17.972090 5098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:17.973335 5098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [term 1 LEADER]: Becoming Leader. State: Replica: 7cdbc752525742088938798d3f513467, State: Running, Role: LEADER
I20260927 05:06:17.973907 5098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbc752525742088938798d3f513467" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41455 } }
I20260927 05:06:17.974293 5093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:17.977090 5099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cdbc752525742088938798d3f513467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbc752525742088938798d3f513467" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41455 } } }
I20260927 05:06:17.977654 5099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:17.978433 5100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cdbc752525742088938798d3f513467. Latest consensus state: current_term: 1 leader_uuid: "7cdbc752525742088938798d3f513467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbc752525742088938798d3f513467" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41455 } } }
I20260927 05:06:17.978853 5100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:17.980006 5105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:17.986414 5105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:17.994961 5105 catalog_manager.cc:1347] Generated new cluster ID: 8528dc2acd564aaf9b9b43e82be3fd8d
I20260927 05:06:17.995100 5105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:18.010144 5105 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:18.010762 5105 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:18.032832 5105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467: Generated new TSK 0
I20260927 05:06:18.033864 5105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:06:18.042513 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41455
--builtin_ntp_servers=127.0.97.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 05:06:18.418113 5117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:18.418462 5117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:18.418623 5117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:18.426221 5117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:18.426393 5117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:18.426581 5117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:18.436786 5117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:18.438243 5117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:18.440275 5117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:18.454434 5122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:18.454497 5125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:18.454416 5123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:18.455492 5117 server_base.cc:1034] running on GCE node
I20260927 05:06:18.456333 5117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:18.457531 5117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:18.458819 5117 hybrid_clock.cc:648] HybridClock initialized: now 1790485578458768 us; error 55 us; skew 500 ppm
I20260927 05:06:18.459205 5117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:06:18.461946 5117 webserver.cc:460] Webserver started at http://127.0.97.1:35925/ using document root <none> and password file <none>
I20260927 05:06:18.462801 5117 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:18.462952 5117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:18.463318 5117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:18.466537 5117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "307340ceeb5a4f6fa34cf4328bac2a23"
format_stamp: "Formatted at 2026-09-27 05:06:18 on dist-test-slave-hgs0"
server_key: "343901216cdaa4cf5635ddb1ee6e9842"
server_key_iv: "2cd76932fe13e64d073243a7d652420e"
server_key_version: "encryptionkey@0"
I20260927 05:06:18.467422 5117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "307340ceeb5a4f6fa34cf4328bac2a23"
format_stamp: "Formatted at 2026-09-27 05:06:18 on dist-test-slave-hgs0"
server_key: "343901216cdaa4cf5635ddb1ee6e9842"
server_key_iv: "2cd76932fe13e64d073243a7d652420e"
server_key_version: "encryptionkey@0"
I20260927 05:06:18.473301 5117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 05:06:18.477526 5131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:18.479346 5117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260927 05:06:18.479581 5117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "307340ceeb5a4f6fa34cf4328bac2a23"
format_stamp: "Formatted at 2026-09-27 05:06:18 on dist-test-slave-hgs0"
server_key: "343901216cdaa4cf5635ddb1ee6e9842"
server_key_iv: "2cd76932fe13e64d073243a7d652420e"
server_key_version: "encryptionkey@0"
I20260927 05:06:18.479876 5117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:18.487432 5117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:18.488593 5117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:18.488950 5117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:18.490095 5117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:18.491825 5117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:18.492028 5117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:18.492183 5117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:18.492285 5117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:18.530599 5117 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34559
I20260927 05:06:18.530642 5243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34559 every 8 connection(s)
I20260927 05:06:18.532408 5117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:18.539537 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 5117
I20260927 05:06:18.539798 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:06:18.540725 388 external_mini_cluster.cc:1428] Setting key 1e132b0b46f08ee57c1ff79bc444b268
I20260927 05:06:18.541951 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:18.535184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51649 (local address 127.0.97.62:41455)
0927 05:06:18.535823 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:18.535827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:18.537185 (+ 1358us) server_negotiation.cc:407] Connection header received
0927 05:06:18.539031 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:18.539037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:18.539115 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:18.539250 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:18.541091 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:18.541096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:18.541100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:18.541119 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:18.541170 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:18.541184 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:18.541334 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:18.541551 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 05:06:18.541630 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":312,"threads_started":1}
I20260927 05:06:18.543040 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:18.535930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41455 (local address 127.0.97.1:51649)
0927 05:06:18.536993 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:18.537039 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:18.538694 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:18.539617 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:18.539627 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:18.539795 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:18.540694 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:06:18.540697 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:18.540739 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:06:18.540868 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:18.541292 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:18.541306 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:18.541317 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 05:06:18.541648 (+ 331us) client_negotiation.cc:240] Negotiation successful
0927 05:06:18.541964 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":315,"threads_started":1}
I20260927 05:06:18.544004 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41455
--builtin_ntp_servers=127.0.97.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 05:06:18.545071 5244 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41455
I20260927 05:06:18.545408 5244 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:18.546036 5244 heartbeater.cc:511] Master 127.0.97.62:41455 requested a full tablet report, sending...
I20260927 05:06:18.548344 5058 ts_manager.cc:194] Registered new tserver with Master: 307340ceeb5a4f6fa34cf4328bac2a23 (127.0.97.1:34559)
W20260927 05:06:18.912693 5248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:18.913093 5248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:18.913264 5248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:18.921077 5248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:18.921240 5248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:18.921425 5248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:18.931543 5248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:18.933044 5248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:18.935109 5248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:18.949896 5256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:18.949811 5253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:18.949857 5254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:18.951267 5248 server_base.cc:1034] running on GCE node
I20260927 05:06:18.952170 5248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:18.953524 5248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:18.954828 5248 hybrid_clock.cc:648] HybridClock initialized: now 1790485578954666 us; error 115 us; skew 500 ppm
I20260927 05:06:18.955607 5248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:06:18.963363 5248 webserver.cc:460] Webserver started at http://127.0.97.2:46483/ using document root <none> and password file <none>
I20260927 05:06:18.964392 5248 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:18.964533 5248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:18.964884 5248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:18.968254 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "6cc880d5b90e4063813c0c725257bf3b"
format_stamp: "Formatted at 2026-09-27 05:06:18 on dist-test-slave-hgs0"
server_key: "80e112dcc8abdd14621fc0091ed42f9a"
server_key_iv: "a254026c421ac8e3186affa852ad75ae"
server_key_version: "encryptionkey@0"
I20260927 05:06:18.969120 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6cc880d5b90e4063813c0c725257bf3b"
format_stamp: "Formatted at 2026-09-27 05:06:18 on dist-test-slave-hgs0"
server_key: "80e112dcc8abdd14621fc0091ed42f9a"
server_key_iv: "a254026c421ac8e3186affa852ad75ae"
server_key_version: "encryptionkey@0"
I20260927 05:06:18.974677 5248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 05:06:18.979139 5262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:18.980922 5248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:18.981159 5248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "6cc880d5b90e4063813c0c725257bf3b"
format_stamp: "Formatted at 2026-09-27 05:06:18 on dist-test-slave-hgs0"
server_key: "80e112dcc8abdd14621fc0091ed42f9a"
server_key_iv: "a254026c421ac8e3186affa852ad75ae"
server_key_version: "encryptionkey@0"
I20260927 05:06:18.981458 5248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:18.988689 5248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:18.989838 5248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:18.990199 5248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:18.991331 5248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:18.993175 5248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:18.993326 5248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:18.993491 5248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:18.993597 5248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:19.035851 5248 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45511
I20260927 05:06:19.035902 5374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45511 every 8 connection(s)
I20260927 05:06:19.037410 5248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:19.043385 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 5248
I20260927 05:06:19.043591 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:06:19.044165 388 external_mini_cluster.cc:1428] Setting key aacb38f6e281f73e4835ea2334fe05b0
I20260927 05:06:19.047842 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.040436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36121 (local address 127.0.97.62:41455)
0927 05:06:19.040664 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.040669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.042455 (+ 1786us) server_negotiation.cc:407] Connection header received
0927 05:06:19.044799 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.044806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.044889 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.045033 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.046852 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.046859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.046865 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.046890 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.046938 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.046960 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.047125 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.047307 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.047384 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 05:06:19.047883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41455
--builtin_ntp_servers=127.0.97.20:34479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 05:06:19.048687 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.041080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41455 (local address 127.0.97.2:36121)
0927 05:06:19.042207 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:19.042303 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:19.044439 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:19.045286 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:19.045299 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:19.045460 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:19.046481 (+ 1021us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:06:19.046485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:19.046541 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:06:19.046652 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:19.047073 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:19.047080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:19.047086 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:06:19.047327 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 05:06:19.047556 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":367,"threads_started":1}
I20260927 05:06:19.050200 5375 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41455
I20260927 05:06:19.050530 5375 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:19.051154 5375 heartbeater.cc:511] Master 127.0.97.62:41455 requested a full tablet report, sending...
I20260927 05:06:19.053020 5058 ts_manager.cc:194] Registered new tserver with Master: 6cc880d5b90e4063813c0c725257bf3b (127.0.97.2:45511)
W20260927 05:06:19.397171 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:19.397543 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:19.397717 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:19.405107 5378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:19.405268 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:19.405465 5378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:19.415306 5378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:19.416752 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:19.418574 5378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:19.432220 5386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:19.432343 5384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:19.432188 5383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:19.433668 5378 server_base.cc:1034] running on GCE node
I20260927 05:06:19.434438 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:19.435472 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:19.436712 5378 hybrid_clock.cc:648] HybridClock initialized: now 1790485579436655 us; error 51 us; skew 500 ppm
I20260927 05:06:19.437084 5378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:06:19.439527 5378 webserver.cc:460] Webserver started at http://127.0.97.3:45595/ using document root <none> and password file <none>
I20260927 05:06:19.440456 5378 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:19.440598 5378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:19.440936 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:19.444160 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2"
format_stamp: "Formatted at 2026-09-27 05:06:19 on dist-test-slave-hgs0"
server_key: "1693fcfdcdbac23ec17ece4f2f294aa6"
server_key_iv: "8a1a92d16414b0b9f8575440932ca012"
server_key_version: "encryptionkey@0"
I20260927 05:06:19.445056 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2"
format_stamp: "Formatted at 2026-09-27 05:06:19 on dist-test-slave-hgs0"
server_key: "1693fcfdcdbac23ec17ece4f2f294aa6"
server_key_iv: "8a1a92d16414b0b9f8575440932ca012"
server_key_version: "encryptionkey@0"
I20260927 05:06:19.450830 5378 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:06:19.454842 5392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:19.456593 5378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:19.456820 5378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2"
format_stamp: "Formatted at 2026-09-27 05:06:19 on dist-test-slave-hgs0"
server_key: "1693fcfdcdbac23ec17ece4f2f294aa6"
server_key_iv: "8a1a92d16414b0b9f8575440932ca012"
server_key_version: "encryptionkey@0"
I20260927 05:06:19.457099 5378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:19.463644 5378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:19.464712 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:19.465039 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:19.466106 5378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:19.467633 5378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:19.467774 5378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:19.467905 5378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:19.468031 5378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 05:06:19.503159 5378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44049
I20260927 05:06:19.503209 5504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44049 every 8 connection(s)
I20260927 05:06:19.504741 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:19.510178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 5378
I20260927 05:06:19.510380 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:06:19.510893 388 external_mini_cluster.cc:1428] Setting key 3cb9d6d7e790e814eb54e4650503608c
I20260927 05:06:19.515054 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.507653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40441 (local address 127.0.97.62:41455)
0927 05:06:19.507868 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.507874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.509589 (+ 1715us) server_negotiation.cc:407] Connection header received
0927 05:06:19.511861 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.511867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.511964 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.512107 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.514165 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.514171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.514174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.514194 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.514238 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.514252 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.514375 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.514681 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.514756 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:06:19.516528 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.508347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41455 (local address 127.0.97.3:40441)
0927 05:06:19.509417 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:19.509458 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:19.511466 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:19.512523 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:19.512538 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:19.512683 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:19.513682 (+ 999us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:06:19.513686 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:19.513753 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:06:19.513935 (+ 182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:19.514404 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:19.514417 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:19.514429 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 05:06:19.514790 (+ 361us) client_negotiation.cc:240] Negotiation successful
0927 05:06:19.515221 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":363,"threads_started":1}
I20260927 05:06:19.518196 5505 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41455
I20260927 05:06:19.518510 5505 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:19.519141 5505 heartbeater.cc:511] Master 127.0.97.62:41455 requested a full tablet report, sending...
I20260927 05:06:19.520743 5058 ts_manager.cc:194] Registered new tserver with Master: db1c5f2f377140ddaaf1cc7fe2d23db2 (127.0.97.3:44049)
I20260927 05:06:19.527624 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:06:19.541980 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.532889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.0.97.62:41455)
0927 05:06:19.533115 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.533124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.533580 (+ 456us) server_negotiation.cc:407] Connection header received
0927 05:06:19.533889 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.533927 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.534025 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.534122 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.541063 (+ 6941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.541070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.541078 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.541113 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.541155 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.541171 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.541388 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.541491 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.541577 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:06:19.551338 5244 heartbeater.cc:503] Master 127.0.97.62:41455 was elected leader, sending a full tablet report...
I20260927 05:06:19.547194 5058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:06:19.554764 5058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:06:19.579216 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.569006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45511 (local address 127.0.0.1:42636)
0927 05:06:19.570754 (+ 1748us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:19.570821 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:19.571036 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:19.573005 (+ 1969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:19.573013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:19.573134 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:19.576984 (+ 3850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:06:19.576991 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:19.577041 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:06:19.577182 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:19.578028 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:19.578049 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:19.578064 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 05:06:19.578334 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 05:06:19.578479 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1357,"thread_start_us":1148,"threads_started":1}
I20260927 05:06:19.580204 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.569186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.0.97.2:45511)
0927 05:06:19.570136 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.570149 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.571217 (+ 1068us) server_negotiation.cc:407] Connection header received
0927 05:06:19.571320 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.571333 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.571696 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.571884 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.577506 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.577523 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.577551 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.577586 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.577697 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.577735 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.577946 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.579536 (+ 1590us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.579680 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":640,"threads_started":1}
I20260927 05:06:19.585198 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.572632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34559 (local address 127.0.0.1:51024)
0927 05:06:19.581969 (+ 9337us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:19.581990 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:19.582102 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:19.582931 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:19.582936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:19.582963 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:19.583920 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:06:19.583927 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:19.583983 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:06:19.584038 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:19.584510 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:19.584515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:19.584522 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 05:06:19.584685 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 05:06:19.584758 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9224,"thread_start_us":8654,"threads_started":1}
I20260927 05:06:19.585232 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.573156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.0.97.1:34559)
0927 05:06:19.574451 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.574466 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.582252 (+ 7786us) server_negotiation.cc:407] Connection header received
0927 05:06:19.582358 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.582370 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.582700 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.582911 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.584224 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.584234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.584253 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.584274 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.584344 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.584373 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.584566 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.584653 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.584739 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":326,"threads_started":1}
I20260927 05:06:19.586303 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.569408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44049 (local address 127.0.0.1:52592)
0927 05:06:19.580985 (+ 11577us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:19.581020 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:19.581170 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:19.583434 (+ 2264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:19.583442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:19.583467 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:19.584878 (+ 1411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:06:19.584888 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:19.584912 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:06:19.584962 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:19.585651 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:19.585659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:19.585665 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:06:19.585813 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 05:06:19.585874 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11368,"thread_start_us":12260,"threads_started":1}
I20260927 05:06:19.587361 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.570007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.0.97.3:44049)
0927 05:06:19.582776 (+ 12769us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.582799 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.582840 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:06:19.582921 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.582930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.583153 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.583338 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.585182 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.585194 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.585215 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.585238 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.585320 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.585345 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.585556 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.586794 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.586905 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12580,"thread_start_us":12415,"threads_started":1}
I20260927 05:06:19.587847 5310 tablet_service.cc:1467] Processing CreateTablet for tablet 30ac1682319448f89c26e4510231d884 (DEFAULT_TABLE table=test-table [id=6b7b01398a884752bb4dca57477a4969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:19.589393 5310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ac1682319448f89c26e4510231d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:19.589600 5179 tablet_service.cc:1467] Processing CreateTablet for tablet 30ac1682319448f89c26e4510231d884 (DEFAULT_TABLE table=test-table [id=6b7b01398a884752bb4dca57477a4969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:19.590806 5179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ac1682319448f89c26e4510231d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:19.591816 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 30ac1682319448f89c26e4510231d884 (DEFAULT_TABLE table=test-table [id=6b7b01398a884752bb4dca57477a4969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:19.594310 5440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ac1682319448f89c26e4510231d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:19.603997 5523 tablet_bootstrap.cc:492] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23: Bootstrap starting.
I20260927 05:06:19.607838 5523 tablet_bootstrap.cc:654] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:19.608935 5524 tablet_bootstrap.cc:492] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: Bootstrap starting.
I20260927 05:06:19.610232 5523 log.cc:824] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:19.612584 5526 tablet_bootstrap.cc:492] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: Bootstrap starting.
I20260927 05:06:19.613492 5524 tablet_bootstrap.cc:654] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:19.613870 5523 tablet_bootstrap.cc:492] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23: No bootstrap required, opened a new log
I20260927 05:06:19.614162 5523 ts_tablet_manager.cc:1397] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 05:06:19.614892 5524 log.cc:824] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:19.616468 5526 tablet_bootstrap.cc:654] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:19.617782 5526 log.cc:824] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:19.618492 5524 tablet_bootstrap.cc:492] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: No bootstrap required, opened a new log
I20260927 05:06:19.618790 5524 ts_tablet_manager.cc:1397] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260927 05:06:19.621484 5526 tablet_bootstrap.cc:492] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: No bootstrap required, opened a new log
I20260927 05:06:19.621858 5526 ts_tablet_manager.cc:1397] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 05:06:19.623319 5523 raft_consensus.cc:348] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 } } peers { permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 } } peers { permanent_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } }
I20260927 05:06:19.623741 5523 raft_consensus.cc:374] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:19.623872 5523 raft_consensus.cc:729] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307340ceeb5a4f6fa34cf4328bac2a23, State: Initialized, Role: FOLLOWER
I20260927 05:06:19.624610 5523 consensus_queue.cc:260] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 } } peers { permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 } } peers { permanent_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } }
I20260927 05:06:19.627671 5523 ts_tablet_manager.cc:1428] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 05:06:19.631079 5524 raft_consensus.cc:348] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 } } peers { permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 } } peers { permanent_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } }
I20260927 05:06:19.631649 5524 raft_consensus.cc:374] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:19.631778 5524 raft_consensus.cc:729] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc880d5b90e4063813c0c725257bf3b, State: Initialized, Role: FOLLOWER
I20260927 05:06:19.632550 5524 consensus_queue.cc:260] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 } } peers { permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 } } peers { permanent_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } }
I20260927 05:06:19.633443 5526 raft_consensus.cc:348] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 } } peers { permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 } } peers { permanent_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } }
I20260927 05:06:19.633945 5526 raft_consensus.cc:374] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:19.633955 5375 heartbeater.cc:503] Master 127.0.97.62:41455 was elected leader, sending a full tablet report...
I20260927 05:06:19.634059 5526 raft_consensus.cc:729] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1c5f2f377140ddaaf1cc7fe2d23db2, State: Initialized, Role: FOLLOWER
I20260927 05:06:19.634723 5524 ts_tablet_manager.cc:1428] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260927 05:06:19.634780 5526 consensus_queue.cc:260] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 } } peers { permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 } } peers { permanent_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } }
I20260927 05:06:19.636097 5505 heartbeater.cc:503] Master 127.0.97.62:41455 was elected leader, sending a full tablet report...
I20260927 05:06:19.636776 5526 ts_tablet_manager.cc:1428] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 05:06:19.691128 5529 raft_consensus.cc:482] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:19.691449 5529 raft_consensus.cc:504] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 } } peers { permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 } } peers { permanent_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } }
I20260927 05:06:19.693163 5529 leader_election.cc:288] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc880d5b90e4063813c0c725257bf3b (127.0.97.2:45511), db1c5f2f377140ddaaf1cc7fe2d23db2 (127.0.97.3:44049)
I20260927 05:06:19.696853 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.693673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41777 (local address 127.0.97.3:44049)
0927 05:06:19.693845 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.693853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.694557 (+ 704us) server_negotiation.cc:407] Connection header received
0927 05:06:19.694753 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.694759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.694823 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.694951 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.696221 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.696230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.696235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.696257 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.696297 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.696310 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.696443 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.696516 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.696582 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:06:19.697031 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.693525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44049 (local address 127.0.97.1:41777)
0927 05:06:19.694405 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:19.694430 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:19.694609 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:19.694997 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:19.695003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:19.695027 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:19.695972 (+ 945us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:06:19.695975 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:19.695995 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:06:19.696050 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:19.696408 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:19.696413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:19.696419 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:06:19.696560 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 05:06:19.696610 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"spinlock_wait_cycles":37120,"thread_start_us":500,"threads_started":1}
I20260927 05:06:19.697041 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.693616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59089 (local address 127.0.97.2:45511)
0927 05:06:19.693811 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.693817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.694555 (+ 738us) server_negotiation.cc:407] Connection header received
0927 05:06:19.694754 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.694759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.694823 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.694951 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.696301 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.696307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.696311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.696333 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.696374 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.696390 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.696505 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.696711 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.696776 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:06:19.697803 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.693574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45511 (local address 127.0.97.1:59089)
0927 05:06:19.694432 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:19.694447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:19.694568 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:19.694998 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:19.695003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:19.695027 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:19.696065 (+ 1038us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:06:19.696068 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:19.696081 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:06:19.696143 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:19.696568 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:19.696572 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:19.696577 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 05:06:19.697291 (+ 714us) client_negotiation.cc:240] Negotiation successful
0927 05:06:19.697348 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":445,"threads_started":1}
I20260927 05:06:19.698114 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ac1682319448f89c26e4510231d884" candidate_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" is_pre_election: true
I20260927 05:06:19.698681 5460 raft_consensus.cc:2454] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 307340ceeb5a4f6fa34cf4328bac2a23 in term 0.
I20260927 05:06:19.698747 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ac1682319448f89c26e4510231d884" candidate_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc880d5b90e4063813c0c725257bf3b" is_pre_election: true
I20260927 05:06:19.699313 5330 raft_consensus.cc:2454] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 307340ceeb5a4f6fa34cf4328bac2a23 in term 0.
I20260927 05:06:19.699380 5134 leader_election.cc:302] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307340ceeb5a4f6fa34cf4328bac2a23, db1c5f2f377140ddaaf1cc7fe2d23db2; no voters:
I20260927 05:06:19.699839 5529 raft_consensus.cc:2781] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:06:19.700023 5529 raft_consensus.cc:482] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:06:19.700155 5529 raft_consensus.cc:3037] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:19.702418 5529 raft_consensus.cc:504] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 } } peers { permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 } } peers { permanent_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } }
I20260927 05:06:19.703042 5529 leader_election.cc:288] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc880d5b90e4063813c0c725257bf3b (127.0.97.2:45511), db1c5f2f377140ddaaf1cc7fe2d23db2 (127.0.97.3:44049)
I20260927 05:06:19.703558 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ac1682319448f89c26e4510231d884" candidate_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc880d5b90e4063813c0c725257bf3b"
I20260927 05:06:19.703565 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ac1682319448f89c26e4510231d884" candidate_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2"
I20260927 05:06:19.703756 5330 raft_consensus.cc:3037] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:19.703775 5460 raft_consensus.cc:3037] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:19.706060 5330 raft_consensus.cc:2454] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307340ceeb5a4f6fa34cf4328bac2a23 in term 1.
I20260927 05:06:19.706075 5460 raft_consensus.cc:2454] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307340ceeb5a4f6fa34cf4328bac2a23 in term 1.
I20260927 05:06:19.706512 5135 leader_election.cc:302] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 307340ceeb5a4f6fa34cf4328bac2a23, 6cc880d5b90e4063813c0c725257bf3b; no voters:
I20260927 05:06:19.706835 5529 raft_consensus.cc:2781] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:19.707263 5529 raft_consensus.cc:686] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 1 LEADER]: Becoming Leader. State: Replica: 307340ceeb5a4f6fa34cf4328bac2a23, State: Running, Role: LEADER
I20260927 05:06:19.707854 5529 consensus_queue.cc:237] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 } } peers { permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 } } peers { permanent_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } }
I20260927 05:06:19.712633 5058 catalog_manager.cc:5495] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 307340ceeb5a4f6fa34cf4328bac2a23 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "307340ceeb5a4f6fa34cf4328bac2a23" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34559 } health_report { overall_health: HEALTHY } } }
I20260927 05:06:19.735332 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.733332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.0.97.1:34559)
0927 05:06:19.733486 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.733491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.733588 (+ 97us) server_negotiation.cc:407] Connection header received
0927 05:06:19.733765 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.733770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.733904 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.734040 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.734883 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.734886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.734889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.734901 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.734927 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.734937 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.735030 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.735096 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.735139 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:06:19.743477 5330 raft_consensus.cc:1261] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b [term 1 FOLLOWER]: Refusing update from remote peer 307340ceeb5a4f6fa34cf4328bac2a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:19.743477 5460 raft_consensus.cc:1261] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2 [term 1 FOLLOWER]: Refusing update from remote peer 307340ceeb5a4f6fa34cf4328bac2a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:19.744439 5534 consensus_queue.cc:1035] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1c5f2f377140ddaaf1cc7fe2d23db2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:19.744757 5529 consensus_queue.cc:1035] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc880d5b90e4063813c0c725257bf3b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:19.755012 5537 mvcc.cc:205] Tried to move back new op lower bound from 7333828934619582464 to 7333828934487687168. Current Snapshot: MvccSnapshot[applied={T|T < 7333828934619582464}]
I20260927 05:06:19.757630 5538 mvcc.cc:205] Tried to move back new op lower bound from 7333828934619582464 to 7333828934487687168. Current Snapshot: MvccSnapshot[applied={T|T < 7333828934619582464}]
I20260927 05:06:19.765517 5539 mvcc.cc:205] Tried to move back new op lower bound from 7333828934619582464 to 7333828934487687168. Current Snapshot: MvccSnapshot[applied={T|T < 7333828934619582464}]
I20260927 05:06:19.770889 5058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:19.771365 5058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:19.776016 5058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7cdbc752525742088938798d3f513467: Sending DeleteTablet for 3 replicas of tablet 30ac1682319448f89c26e4510231d884
I20260927 05:06:19.777522 5179 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ac1682319448f89c26e4510231d884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:19 UTC) from {username='slave'} at 127.0.0.1:51024
I20260927 05:06:19.777508 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ac1682319448f89c26e4510231d884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:19 UTC) from {username='slave'} at 127.0.0.1:52592
I20260927 05:06:19.778471 5546 tablet_replica.cc:330] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23: stopping tablet replica
I20260927 05:06:19.778524 5310 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ac1682319448f89c26e4510231d884 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:19 UTC) from {username='slave'} at 127.0.0.1:42636
I20260927 05:06:19.778751 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 5117
I20260927 05:06:19.779127 5546 raft_consensus.cc:2229] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:06:19.779975 5546 raft_consensus.cc:2258] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:19.782437 5546 ts_tablet_manager.cc:1905] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:19.783087 5547 tablet_replica.cc:330] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: stopping tablet replica
I20260927 05:06:19.783591 5547 raft_consensus.cc:2229] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:19.786572 5547 raft_consensus.cc:2258] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:19.789139 5547 ts_tablet_manager.cc:1905] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:19.791199 5548 tablet_replica.cc:330] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: stopping tablet replica
I20260927 05:06:19.791920 5548 raft_consensus.cc:2229] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:19.792474 5548 raft_consensus.cc:2258] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:19.794575 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.782103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.0.97.1:34559)
0927 05:06:19.782333 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.782343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.791572 (+ 9229us) server_negotiation.cc:407] Connection header received
0927 05:06:19.792067 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.792073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.792142 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.792231 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.793675 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.793681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.793687 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.793713 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.793751 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.793766 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.793880 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.794143 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.794220 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 05:06:19.795826 5239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:19.798170 5548 ts_tablet_manager.cc:1905] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:19.800118 5547 ts_tablet_manager.cc:1918] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:19.800549 5547 log.cc:1196] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/30ac1682319448f89c26e4510231d884
I20260927 05:06:19.801149 5547 ts_tablet_manager.cc:1939] T 30ac1682319448f89c26e4510231d884 P db1c5f2f377140ddaaf1cc7fe2d23db2: Deleting consensus metadata
I20260927 05:06:19.803578 5044 catalog_manager.cc:4841] TS db1c5f2f377140ddaaf1cc7fe2d23db2 (127.0.97.3:44049): tablet 30ac1682319448f89c26e4510231d884 (table test-table [id=6b7b01398a884752bb4dca57477a4969]) successfully deleted
I20260927 05:06:19.805876 5548 ts_tablet_manager.cc:1918] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:19.806100 5548 log.cc:1196] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/30ac1682319448f89c26e4510231d884
I20260927 05:06:19.806646 5548 ts_tablet_manager.cc:1939] T 30ac1682319448f89c26e4510231d884 P 6cc880d5b90e4063813c0c725257bf3b: Deleting consensus metadata
I20260927 05:06:19.808782 5045 catalog_manager.cc:4841] TS 6cc880d5b90e4063813c0c725257bf3b (127.0.97.2:45511): tablet 30ac1682319448f89c26e4510231d884 (table test-table [id=6b7b01398a884752bb4dca57477a4969]) successfully deleted
I20260927 05:06:19.900323 5546 ts_tablet_manager.cc:1918] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:19.900655 5546 log.cc:1196] T 30ac1682319448f89c26e4510231d884 P 307340ceeb5a4f6fa34cf4328bac2a23: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/30ac1682319448f89c26e4510231d884
I20260927 05:06:19.901108 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 5117
W20260927 05:06:19.919574 5045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:34559 (error 108)
I20260927 05:06:19.919700 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 5248
I20260927 05:06:19.921674 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.921109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34559 (local address 127.0.0.1:51058)
0927 05:06:19.921324 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:19.921500 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 05:06:19.921976 5045 catalog_manager.cc:4584] TS 307340ceeb5a4f6fa34cf4328bac2a23 (127.0.97.1:34559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30ac1682319448f89c26e4510231d884: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34559: connect: Connection refused (error 111)
I20260927 05:06:19.923156 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.920818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.0.97.2:45511)
0927 05:06:19.921031 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:19.921038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:19.921056 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:06:19.921121 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:19.921125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:19.921261 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:19.921376 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:19.922439 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:19.922445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:19.922450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:19.922473 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:19.922512 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:19.922525 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:19.922662 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:19.922793 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 05:06:19.922859 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:06:19.923882 5370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:19.975720 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.975262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34559 (local address 127.0.0.1:51068)
0927 05:06:19.975455 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:19.975554 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 05:06:19.976833 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:19.976485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34559 (local address 127.0.0.1:51070)
0927 05:06:19.976620 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:19.976699 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 05:06:20.021091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 5248
I20260927 05:06:20.037580 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:20.037088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34559 (local address 127.0.0.1:51074)
0927 05:06:20.037309 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:20.037413 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 05:06:20.038764 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:20.038332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34559 (local address 127.0.0.1:51084)
0927 05:06:20.038496 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:20.038578 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 05:06:20.039080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 5378
I20260927 05:06:20.041898 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:20.039820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.0.97.3:44049)
0927 05:06:20.040040 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:20.040046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:20.040068 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:06:20.040175 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:20.040181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:20.040277 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:20.040395 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:20.041223 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:20.041230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:20.041234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:20.041253 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:20.041294 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:20.041305 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:20.041465 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:20.041546 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 05:06:20.041606 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:06:20.042716 5500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:20.120105 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:20.119282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34559 (local address 127.0.0.1:51094)
0927 05:06:20.119563 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:20.119732 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 05:06:20.121377 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:20.120887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34559 (local address 127.0.0.1:51100)
0927 05:06:20.121126 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:20.121218 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:34559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 05:06:20.136485 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 5378
I20260927 05:06:20.154754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 5027
I20260927 05:06:20.156203 5088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:20.269356 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 5027
2026-09-27T05:06:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2860 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 05:06:20.313174 388 mini_oidc.cc:152] Starting JWKS server
I20260927 05:06:20.316048 388 webserver.cc:460] Webserver started at https://127.0.0.1:40861/ using document root <none> and password file <none>
I20260927 05:06:20.316462 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 05:06:20.317230 388 webserver.cc:460] Webserver started at http://127.0.0.1:37755/ using document root <none> and password file <none>
2026-09-27T05:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:20Z Disabled control of system clock
I20260927 05:06:20.346460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34711
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34711
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40861/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/testchainca.cert with env {}
W20260927 05:06:20.706722 5560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:20.707042 5560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:20.707139 5560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:20.714646 5560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:20.714794 5560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:20.714838 5560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:20.714903 5560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:20.714956 5560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 05:06:20.715020 5560 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 05:06:20.715067 5560 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40861/jwks.json
I20260927 05:06:20.725834 5560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40861/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:20.727361 5560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:20.728967 5560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:20.743198 5566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:20.743311 5565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:20.743311 5568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:20.744692 5560 server_base.cc:1034] running on GCE node
I20260927 05:06:20.745592 5560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:20.747193 5560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:20.748437 5560 hybrid_clock.cc:648] HybridClock initialized: now 1790485580748403 us; error 48 us; skew 500 ppm
I20260927 05:06:20.748811 5560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:06:20.758010 5560 webserver.cc:460] Webserver started at http://127.0.97.62:44719/ using document root <none> and password file <none>
I20260927 05:06:20.758816 5560 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:20.758922 5560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:20.759198 5560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:20.762426 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "f2029074d0184bc4b15e156e1f3c3486"
format_stamp: "Formatted at 2026-09-27 05:06:20 on dist-test-slave-hgs0"
server_key: "e00d84ecf792538f0469c536ceaad1f2"
server_key_iv: "f289146029628a8d6e4c08eb89117a4e"
server_key_version: "encryptionkey@0"
I20260927 05:06:20.763240 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "f2029074d0184bc4b15e156e1f3c3486"
format_stamp: "Formatted at 2026-09-27 05:06:20 on dist-test-slave-hgs0"
server_key: "e00d84ecf792538f0469c536ceaad1f2"
server_key_iv: "f289146029628a8d6e4c08eb89117a4e"
server_key_version: "encryptionkey@0"
I20260927 05:06:20.769098 5560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260927 05:06:20.773283 5574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:20.775084 5560 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260927 05:06:20.775321 5560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "f2029074d0184bc4b15e156e1f3c3486"
format_stamp: "Formatted at 2026-09-27 05:06:20 on dist-test-slave-hgs0"
server_key: "e00d84ecf792538f0469c536ceaad1f2"
server_key_iv: "f289146029628a8d6e4c08eb89117a4e"
server_key_version: "encryptionkey@0"
I20260927 05:06:20.775591 5560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:20.797689 5560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:20.798614 5560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:20.798923 5560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:20.818222 5560 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34711
I20260927 05:06:20.818310 5625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34711 every 8 connection(s)
I20260927 05:06:20.819871 5560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:20.825203 5626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:20.828550 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 5560
I20260927 05:06:20.828852 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:20.829499 388 external_mini_cluster.cc:1428] Setting key ca27aec6ddb879a52e43ef1ce480fbd8
I20260927 05:06:20.840198 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486: Bootstrap starting.
I20260927 05:06:20.844960 5626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:20.846473 5626 log.cc:824] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:20.847147 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:20.831835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.0.97.62:34711)
0927 05:06:20.833072 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:20.833086 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:20.833137 (+ 51us) server_negotiation.cc:407] Connection header received
0927 05:06:20.835053 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:20.835081 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:20.835714 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:20.836277 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:20.837311 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:20.838816 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:20.841135 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:20.842033 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:20.842247 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:20.843847 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:20.843857 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:20.843875 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:20.844665 (+ 790us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:20.844828 (+ 163us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:20.844856 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:20.845038 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:20.845311 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 05:06:20.845742 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":421,"threads_started":1}
I20260927 05:06:20.850075 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486: No bootstrap required, opened a new log
I20260927 05:06:20.851653 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:06:20.858986 5626 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2029074d0184bc4b15e156e1f3c3486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34711 } }
I20260927 05:06:20.859472 5626 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:20.859593 5626 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2029074d0184bc4b15e156e1f3c3486, State: Initialized, Role: FOLLOWER
I20260927 05:06:20.860332 5626 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2029074d0184bc4b15e156e1f3c3486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34711 } }
I20260927 05:06:20.860594 5626 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:20.860752 5626 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:20.860947 5626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:20.863269 5626 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2029074d0184bc4b15e156e1f3c3486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34711 } }
I20260927 05:06:20.863894 5626 leader_election.cc:302] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2029074d0184bc4b15e156e1f3c3486; no voters:
I20260927 05:06:20.864616 5626 leader_election.cc:288] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:20.864825 5631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:20.867018 5626 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:20.867007 5631 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [term 1 LEADER]: Becoming Leader. State: Replica: f2029074d0184bc4b15e156e1f3c3486, State: Running, Role: LEADER
I20260927 05:06:20.867630 5631 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2029074d0184bc4b15e156e1f3c3486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34711 } }
I20260927 05:06:20.871294 5632 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2029074d0184bc4b15e156e1f3c3486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2029074d0184bc4b15e156e1f3c3486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34711 } } }
I20260927 05:06:20.871359 5633 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2029074d0184bc4b15e156e1f3c3486. Latest consensus state: current_term: 1 leader_uuid: "f2029074d0184bc4b15e156e1f3c3486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2029074d0184bc4b15e156e1f3c3486" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34711 } } }
I20260927 05:06:20.871769 5632 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:20.871769 5633 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:20.874179 5641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:20.880595 5641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:20.890969 5641 catalog_manager.cc:1347] Generated new cluster ID: 1f5f3cdb6b814331b3553277fa709cb7
I20260927 05:06:20.891136 5641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:20.917428 5641 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:20.918978 5641 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:20.928776 5641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2029074d0184bc4b15e156e1f3c3486: Generated new TSK 0
I20260927 05:06:20.929804 5641 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 05:06:20.972566 5656 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 05:06:20.972796 5656 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 05:06:20.974059 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:20.949694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.0.97.62:34711)
0927 05:06:20.949873 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:20.949878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:20.950354 (+ 476us) server_negotiation.cc:407] Connection header received
0927 05:06:20.950651 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:20.950658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:20.950753 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:20.950934 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0927 05:06:20.951461 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:20.952663 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:20.954074 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:20.954768 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:20.954906 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:20.955455 (+ 549us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 05:06:20.973264 (+ 17809us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 05:06:20.973712 (+ 448us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45830: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 05:06:20.974200 5629 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45830: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 05:06:20.975683 5655 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:20.949529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34711 (local address 127.0.0.1:45830)
0927 05:06:20.950235 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:20.950258 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:20.950417 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:20.950966 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:20.950970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:20.950979 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 05:06:20.951310 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:20.951318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:20.952812 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:20.952815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:20.953933 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:20.953943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:20.954898 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:20.954955 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:20.955078 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:20.955138 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 05:06:20.973595 (+ 18457us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 05:06:20.973970 (+ 375us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34711: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":381,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:34711: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 05:06:20.977510 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 5560
I20260927 05:06:20.978710 5621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:21.081941 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 5560
2026-09-27T05:06:21Z chronyd exiting
I20260927 05:06:21.132998 388 test_util.cc:175] -----------------------------------------------
I20260927 05:06:21.133139 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790485535376807-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (824 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 05:06:21.136848 388 mini_oidc.cc:152] Starting JWKS server
I20260927 05:06:21.138720 388 webserver.cc:460] Webserver started at https://127.0.0.1:46469/ using document root <none> and password file <none>
I20260927 05:06:21.138958 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 05:06:21.139848 388 webserver.cc:460] Webserver started at http://127.0.0.1:33523/ using document root <none> and password file <none>
2026-09-27T05:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:21Z Disabled control of system clock
I20260927 05:06:21.169862 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42149
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42149
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46469/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/testchainca.cert with env {}
W20260927 05:06:21.524986 5663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:21.525501 5663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:21.525687 5663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:21.533740 5663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:21.533929 5663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:21.534029 5663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:21.534142 5663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:21.534228 5663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 05:06:21.534327 5663 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 05:06:21.534406 5663 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46469/jwks.json
I20260927 05:06:21.545558 5663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46469/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:21.547245 5663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:21.548799 5663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:21.562276 5668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:21.562211 5671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:21.562255 5669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:21.563195 5663 server_base.cc:1034] running on GCE node
I20260927 05:06:21.564107 5663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:21.565621 5663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:21.566834 5663 hybrid_clock.cc:648] HybridClock initialized: now 1790485581566787 us; error 34 us; skew 500 ppm
I20260927 05:06:21.567245 5663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:06:21.569900 5663 webserver.cc:460] Webserver started at http://127.0.97.62:39403/ using document root <none> and password file <none>
I20260927 05:06:21.570787 5663 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:21.570928 5663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:21.571318 5663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:21.574687 5663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "669d1e5c7f064c49952f7b42e9a51485"
format_stamp: "Formatted at 2026-09-27 05:06:21 on dist-test-slave-hgs0"
server_key: "14315c08f701a51d95301b83643ff29e"
server_key_iv: "ee8913e83918b73fc079a3b99afaf512"
server_key_version: "encryptionkey@0"
I20260927 05:06:21.575589 5663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "669d1e5c7f064c49952f7b42e9a51485"
format_stamp: "Formatted at 2026-09-27 05:06:21 on dist-test-slave-hgs0"
server_key: "14315c08f701a51d95301b83643ff29e"
server_key_iv: "ee8913e83918b73fc079a3b99afaf512"
server_key_version: "encryptionkey@0"
I20260927 05:06:21.581372 5663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:06:21.585493 5677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:21.587369 5663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:06:21.587641 5663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "669d1e5c7f064c49952f7b42e9a51485"
format_stamp: "Formatted at 2026-09-27 05:06:21 on dist-test-slave-hgs0"
server_key: "14315c08f701a51d95301b83643ff29e"
server_key_iv: "ee8913e83918b73fc079a3b99afaf512"
server_key_version: "encryptionkey@0"
I20260927 05:06:21.587934 5663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:21.622352 5663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:21.623318 5663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:21.623636 5663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:21.643713 5663 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42149
I20260927 05:06:21.643733 5728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42149 every 8 connection(s)
I20260927 05:06:21.645259 5663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:21.649610 5729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:21.652331 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 5663
I20260927 05:06:21.652552 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:21.653270 388 external_mini_cluster.cc:1428] Setting key 3e1b7622dd2b8f37bf1a31a94e15d8b4
I20260927 05:06:21.664350 5729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485: Bootstrap starting.
I20260927 05:06:21.668429 5729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:21.669785 5729 log.cc:824] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:21.670248 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:21.655645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.0.97.62:42149)
0927 05:06:21.657005 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:21.657031 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:21.657089 (+ 58us) server_negotiation.cc:407] Connection header received
0927 05:06:21.659566 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:21.659607 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:21.660371 (+ 764us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:21.660888 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:21.661498 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:21.663466 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:21.664878 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:21.665909 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:21.666136 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:21.667208 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:21.667218 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:21.667254 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:21.668004 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:21.668135 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:21.668164 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:21.668301 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:21.668551 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 05:06:21.668938 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":434,"threads_started":1}
I20260927 05:06:21.673128 5729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485: No bootstrap required, opened a new log
I20260927 05:06:21.674259 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:06:21.681212 5729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669d1e5c7f064c49952f7b42e9a51485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42149 } }
I20260927 05:06:21.681640 5729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:21.681756 5729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669d1e5c7f064c49952f7b42e9a51485, State: Initialized, Role: FOLLOWER
I20260927 05:06:21.682404 5729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669d1e5c7f064c49952f7b42e9a51485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42149 } }
I20260927 05:06:21.682662 5729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:21.682816 5729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:21.683000 5729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:21.685741 5729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669d1e5c7f064c49952f7b42e9a51485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42149 } }
I20260927 05:06:21.686287 5729 leader_election.cc:302] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 669d1e5c7f064c49952f7b42e9a51485; no voters:
I20260927 05:06:21.686888 5729 leader_election.cc:288] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:21.687036 5734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:21.687777 5734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [term 1 LEADER]: Becoming Leader. State: Replica: 669d1e5c7f064c49952f7b42e9a51485, State: Running, Role: LEADER
I20260927 05:06:21.688349 5734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669d1e5c7f064c49952f7b42e9a51485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42149 } }
I20260927 05:06:21.689088 5729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:21.691722 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 669d1e5c7f064c49952f7b42e9a51485. Latest consensus state: current_term: 1 leader_uuid: "669d1e5c7f064c49952f7b42e9a51485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669d1e5c7f064c49952f7b42e9a51485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42149 } } }
I20260927 05:06:21.692334 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:21.692396 5735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "669d1e5c7f064c49952f7b42e9a51485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669d1e5c7f064c49952f7b42e9a51485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42149 } } }
I20260927 05:06:21.692780 5735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:21.697134 5743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:21.702682 5743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:21.712608 5743 catalog_manager.cc:1347] Generated new cluster ID: 5ac5d331c1ab4563872f1db47a87e155
I20260927 05:06:21.712744 5743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:21.750429 5743 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:21.752045 5743 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:21.760574 5743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 669d1e5c7f064c49952f7b42e9a51485: Generated new TSK 0
I20260927 05:06:21.761559 5743 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 05:06:21.811158 5759 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 05:06:21.811314 5759 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 05:06:21.812383 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:21.782320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.0.97.62:42149)
0927 05:06:21.782561 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:21.782569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:21.782992 (+ 423us) server_negotiation.cc:407] Connection header received
0927 05:06:21.783279 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:21.783284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:21.783368 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:21.783509 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0927 05:06:21.784169 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:21.784989 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:21.787362 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:21.787900 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:21.788094 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:21.788727 (+ 633us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 05:06:21.811717 (+ 22990us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 05:06:21.812144 (+ 427us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43926: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 05:06:21.812537 5732 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43926: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 05:06:21.812747 5758 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:21.782109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42149 (local address 127.0.0.1:43926)
0927 05:06:21.782849 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:21.782877 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:21.783056 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:21.783563 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:21.783568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:21.783583 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0927 05:06:21.783925 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:21.783967 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:21.785166 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:21.785173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:21.787144 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:21.787164 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:21.788096 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:21.788155 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:21.788391 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:21.788446 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 05:06:21.812033 (+ 23587us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 05:06:21.812344 (+ 311us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42149: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46469/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":387,"threads_started":1}
I20260927 05:06:21.814596 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 5663
I20260927 05:06:21.816020 5724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:21.920382 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 5663
2026-09-27T05:06:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (844 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 05:06:21.981925 388 mini_oidc.cc:152] Starting JWKS server
I20260927 05:06:21.984134 388 webserver.cc:460] Webserver started at https://127.0.0.1:33965/ using document root <none> and password file <none>
I20260927 05:06:21.984414 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 05:06:21.985137 388 webserver.cc:460] Webserver started at http://127.0.0.1:40771/ using document root <none> and password file <none>
2026-09-27T05:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:21Z Disabled control of system clock
I20260927 05:06:22.014875 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41845
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41845
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33965/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 05:06:22.376856 5766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:22.377192 5766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:22.377288 5766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:22.385686 5766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:22.385807 5766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:22.385844 5766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:22.385895 5766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:22.385941 5766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 05:06:22.386001 5766 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 05:06:22.386040 5766 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33965/jwks.json
I20260927 05:06:22.395501 5766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33965/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:22.396833 5766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:22.398583 5766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:22.410907 5774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:22.410840 5772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:22.410960 5771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:22.411685 5766 server_base.cc:1034] running on GCE node
I20260927 05:06:22.412537 5766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:22.414068 5766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:22.415272 5766 hybrid_clock.cc:648] HybridClock initialized: now 1790485582415236 us; error 40 us; skew 500 ppm
I20260927 05:06:22.415647 5766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:06:22.418085 5766 webserver.cc:460] Webserver started at http://127.0.97.62:38423/ using document root <none> and password file <none>
I20260927 05:06:22.418838 5766 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:22.418955 5766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:22.419317 5766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:22.422585 5766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "6582d3e37d3649498e467b223fc5967e"
format_stamp: "Formatted at 2026-09-27 05:06:22 on dist-test-slave-hgs0"
server_key: "3a46ca910c76e6ef532d3c1dc45a1f35"
server_key_iv: "78e9d9add4f1c0b6d0bed46971bab363"
server_key_version: "encryptionkey@0"
I20260927 05:06:22.423486 5766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "6582d3e37d3649498e467b223fc5967e"
format_stamp: "Formatted at 2026-09-27 05:06:22 on dist-test-slave-hgs0"
server_key: "3a46ca910c76e6ef532d3c1dc45a1f35"
server_key_iv: "78e9d9add4f1c0b6d0bed46971bab363"
server_key_version: "encryptionkey@0"
I20260927 05:06:22.429445 5766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 05:06:22.433269 5780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:22.434777 5766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:06:22.435003 5766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "6582d3e37d3649498e467b223fc5967e"
format_stamp: "Formatted at 2026-09-27 05:06:22 on dist-test-slave-hgs0"
server_key: "3a46ca910c76e6ef532d3c1dc45a1f35"
server_key_iv: "78e9d9add4f1c0b6d0bed46971bab363"
server_key_version: "encryptionkey@0"
I20260927 05:06:22.435237 5766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:22.449448 5766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:22.450341 5766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:22.450646 5766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:22.469734 5766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41845
I20260927 05:06:22.469754 5831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41845 every 8 connection(s)
I20260927 05:06:22.471194 5766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:22.475225 5832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:22.476500 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 5766
I20260927 05:06:22.476742 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:22.477560 388 external_mini_cluster.cc:1428] Setting key 106ce0bb265cccc579071637ee70351f
I20260927 05:06:22.489943 5832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e: Bootstrap starting.
I20260927 05:06:22.493674 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:22.479825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.0.97.62:41845)
0927 05:06:22.481265 (+ 1440us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:22.481282 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:22.481330 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:06:22.482998 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:22.483025 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:22.483602 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:22.484100 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:22.484750 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:22.486232 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:22.487745 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:22.488657 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:22.488938 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:22.490333 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:22.490345 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:22.490368 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:22.491196 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:22.491310 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:22.491341 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:22.491492 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:22.491758 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 05:06:22.492249 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":553,"threads_started":1}
I20260927 05:06:22.494445 5832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:22.495718 5832 log.cc:824] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:22.498076 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:06:22.498976 5832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e: No bootstrap required, opened a new log
I20260927 05:06:22.506855 5832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582d3e37d3649498e467b223fc5967e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41845 } }
I20260927 05:06:22.507292 5832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:22.507403 5832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6582d3e37d3649498e467b223fc5967e, State: Initialized, Role: FOLLOWER
I20260927 05:06:22.508121 5832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582d3e37d3649498e467b223fc5967e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41845 } }
I20260927 05:06:22.508371 5832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:22.508520 5832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:22.508697 5832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:22.511144 5832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582d3e37d3649498e467b223fc5967e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41845 } }
I20260927 05:06:22.511667 5832 leader_election.cc:302] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6582d3e37d3649498e467b223fc5967e; no voters:
I20260927 05:06:22.512303 5832 leader_election.cc:288] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:22.512463 5837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:22.513180 5837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [term 1 LEADER]: Becoming Leader. State: Replica: 6582d3e37d3649498e467b223fc5967e, State: Running, Role: LEADER
I20260927 05:06:22.513697 5837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582d3e37d3649498e467b223fc5967e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41845 } }
I20260927 05:06:22.514475 5832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:22.516770 5839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6582d3e37d3649498e467b223fc5967e. Latest consensus state: current_term: 1 leader_uuid: "6582d3e37d3649498e467b223fc5967e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582d3e37d3649498e467b223fc5967e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41845 } } }
I20260927 05:06:22.517226 5839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:22.518193 5845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:22.518128 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6582d3e37d3649498e467b223fc5967e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6582d3e37d3649498e467b223fc5967e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41845 } } }
I20260927 05:06:22.519304 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:22.524739 5845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:22.535583 5845 catalog_manager.cc:1347] Generated new cluster ID: ddf18a3896654bf8afb1867daf5d2dbb
I20260927 05:06:22.535718 5845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:22.555558 5845 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:22.557520 5845 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:22.566766 5845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6582d3e37d3649498e467b223fc5967e: Generated new TSK 0
I20260927 05:06:22.567984 5845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:06:22.595512 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:22.586134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.0.97.62:41845)
0927 05:06:22.586351 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:22.586359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:22.586852 (+ 493us) server_negotiation.cc:407] Connection header received
0927 05:06:22.587101 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:22.587107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:22.587207 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:22.587408 (+ 201us) server_negotiation.cc:226] Negotiated authn=JWT
0927 05:06:22.587858 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:22.589052 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:22.590500 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:22.591254 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:22.591402 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:22.591985 (+ 583us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 05:06:22.594721 (+ 2736us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33965/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 05:06:22.595141 (+ 420us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56940: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33965/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 05:06:22.595660 5835 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56940: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33965/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 05:06:22.597432 5861 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:22.586004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41845 (local address 127.0.0.1:56940)
0927 05:06:22.586740 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:22.586757 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:22.586894 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:22.587419 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:22.587423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:22.587432 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 05:06:22.587696 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:22.587703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:22.589201 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:22.589205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:22.590343 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:22.590356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:22.591392 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:22.591461 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:22.591614 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:22.591651 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 05:06:22.594987 (+ 3336us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33965/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 05:06:22.596967 (+ 1980us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33965/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":1574,"thread_start_us":406,"threads_started":1}
I20260927 05:06:22.599335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 5766
I20260927 05:06:22.600803 5827 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 05:06:22.611419 5862 webserver.cc:533] Webserver: sslize failed:
E20260927 05:06:22.611595 5862 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 05:06:22.697497 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 5766
2026-09-27T05:06:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (774 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T05:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:22Z Disabled control of system clock
I20260927 05:06:22.788252 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34433
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34433
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/testchainca.cert with env {}
W20260927 05:06:23.131773 5867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:23.132201 5867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:23.132294 5867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:23.139708 5867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:23.139832 5867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:23.139870 5867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:23.140003 5867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:23.140075 5867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 05:06:23.140137 5867 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 05:06:23.140180 5867 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 05:06:23.151835 5867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34433
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:23.153265 5867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:23.154556 5867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:23.167210 5872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:23.167089 5875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:23.167125 5873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:23.168594 5867 server_base.cc:1034] running on GCE node
I20260927 05:06:23.169553 5867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:23.171310 5867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:23.172521 5867 hybrid_clock.cc:648] HybridClock initialized: now 1790485583172476 us; error 35 us; skew 500 ppm
I20260927 05:06:23.172914 5867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:06:23.175364 5867 webserver.cc:460] Webserver started at http://127.0.97.62:34243/ using document root <none> and password file <none>
I20260927 05:06:23.176172 5867 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:23.176273 5867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:23.176539 5867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:23.179473 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "3ccb8079c8f8481894efd77ed894fb12"
format_stamp: "Formatted at 2026-09-27 05:06:23 on dist-test-slave-hgs0"
server_key: "2fb849375c37f0242cd7cb8826c1dc37"
server_key_iv: "8cc29ecd0b86b3550725f39df12889bd"
server_key_version: "encryptionkey@0"
I20260927 05:06:23.180315 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "3ccb8079c8f8481894efd77ed894fb12"
format_stamp: "Formatted at 2026-09-27 05:06:23 on dist-test-slave-hgs0"
server_key: "2fb849375c37f0242cd7cb8826c1dc37"
server_key_iv: "8cc29ecd0b86b3550725f39df12889bd"
server_key_version: "encryptionkey@0"
I20260927 05:06:23.185757 5867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:06:23.189394 5881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:23.190837 5867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:06:23.190991 5867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "3ccb8079c8f8481894efd77ed894fb12"
format_stamp: "Formatted at 2026-09-27 05:06:23 on dist-test-slave-hgs0"
server_key: "2fb849375c37f0242cd7cb8826c1dc37"
server_key_iv: "8cc29ecd0b86b3550725f39df12889bd"
server_key_version: "encryptionkey@0"
I20260927 05:06:23.191179 5867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:23.209121 5867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:23.210032 5867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:23.210290 5867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:23.228857 5867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34433
I20260927 05:06:23.228834 5932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34433 every 8 connection(s)
I20260927 05:06:23.230230 5867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:23.234113 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:23.239288 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 5867
I20260927 05:06:23.239480 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:23.240152 388 external_mini_cluster.cc:1428] Setting key 0592631d761dda0e06fde1a20cebf61d
I20260927 05:06:23.245157 5933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12: Bootstrap starting.
I20260927 05:06:23.248793 5933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:23.250066 5933 log.cc:824] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:23.253491 5933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12: No bootstrap required, opened a new log
I20260927 05:06:23.254609 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:23.242040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.0.97.62:34433)
0927 05:06:23.243142 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:23.243153 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:23.243194 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:06:23.244965 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:23.244990 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:23.245527 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:23.245870 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:23.246496 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:23.247800 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:23.249636 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:23.250401 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:23.250568 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:23.251850 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:23.251862 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:23.251883 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:06:23.252594 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:23.252696 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:06:23.252722 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:23.252842 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:23.253033 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 05:06:23.253362 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":418,"threads_started":1}
I20260927 05:06:23.258477 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:06:23.261963 5933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccb8079c8f8481894efd77ed894fb12" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34433 } }
I20260927 05:06:23.262323 5933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:23.262434 5933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccb8079c8f8481894efd77ed894fb12, State: Initialized, Role: FOLLOWER
I20260927 05:06:23.263042 5933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccb8079c8f8481894efd77ed894fb12" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34433 } }
I20260927 05:06:23.263269 5933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:23.263420 5933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:23.263582 5933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:23.266105 5933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccb8079c8f8481894efd77ed894fb12" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34433 } }
I20260927 05:06:23.266619 5933 leader_election.cc:302] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ccb8079c8f8481894efd77ed894fb12; no voters:
I20260927 05:06:23.267256 5933 leader_election.cc:288] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:23.267472 5938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:23.268354 5938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [term 1 LEADER]: Becoming Leader. State: Replica: 3ccb8079c8f8481894efd77ed894fb12, State: Running, Role: LEADER
I20260927 05:06:23.268893 5938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccb8079c8f8481894efd77ed894fb12" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34433 } }
I20260927 05:06:23.269647 5933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:23.271804 5940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ccb8079c8f8481894efd77ed894fb12. Latest consensus state: current_term: 1 leader_uuid: "3ccb8079c8f8481894efd77ed894fb12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccb8079c8f8481894efd77ed894fb12" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34433 } } }
I20260927 05:06:23.272298 5940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:23.272130 5939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ccb8079c8f8481894efd77ed894fb12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccb8079c8f8481894efd77ed894fb12" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34433 } } }
I20260927 05:06:23.272434 5939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:23.274078 5947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:23.278604 5947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:23.287053 5947 catalog_manager.cc:1347] Generated new cluster ID: 89c44f574dea4a6289e21c7a04fb1152
I20260927 05:06:23.287189 5947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:23.299463 5947 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:23.301553 5947 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:23.313330 5947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ccb8079c8f8481894efd77ed894fb12: Generated new TSK 0
I20260927 05:06:23.314443 5947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:06:23.757085 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:23.325760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.0.97.62:34433)
0927 05:06:23.325974 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:23.325983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:23.326441 (+ 458us) server_negotiation.cc:407] Connection header received
0927 05:06:23.326694 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:23.326698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:23.326775 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:23.326959 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0927 05:06:23.327413 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:23.328217 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:23.329591 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:23.330076 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:23.330216 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:23.330695 (+ 479us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 05:06:23.756200 (+425505us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 05:06:23.756777 (+ 577us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49098: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 05:06:23.757274 5936 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49098: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 05:06:23.757603 5962 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:23.325754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34433 (local address 127.0.0.1:49098)
0927 05:06:23.326326 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:23.326342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:23.326510 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:23.326969 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:23.326973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:23.326985 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 05:06:23.327258 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:23.327267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:23.328370 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:23.328380 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:23.329442 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:23.329453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:23.330199 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:23.330256 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:23.330406 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:23.330440 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 05:06:23.756613 (+426173us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 05:06:23.756961 (+ 348us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34433: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":308,"threads_started":1}
I20260927 05:06:23.759421 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 5867
I20260927 05:06:23.760560 5928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:23.863718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 5867
2026-09-27T05:06:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1145 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (411 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1461 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:06:25 dist-test-slave-hgs0 krb5kdc[5978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:06:25 dist-test-slave-hgs0 krb5kdc[5978](info): set up 2 sockets
Sep 27 05:06:25 dist-test-slave-hgs0 krb5kdc[5978](info): commencing operation
krb5kdc: starting...
W20260927 05:06:26.824666 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:06:26 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:06:27.027688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46627
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:27.358120 5991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:27.358476 5991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:27.358640 5991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:27.365773 5991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:27.365921 5991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:27.366029 5991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:27.366132 5991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:27.366216 5991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:27.376060 5991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:27.377494 5991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:27.378793 5991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:27.390194 5996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:27.390308 5997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:27.390352 5999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:27.391239 5991 server_base.cc:1034] running on GCE node
I20260927 05:06:27.392030 5991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:27.393369 5991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:27.394572 5991 hybrid_clock.cc:648] HybridClock initialized: now 1790485587394533 us; error 31 us; skew 500 ppm
I20260927 05:06:27.394920 5991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:27 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:27.398016 5991 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:06:27.399343 5991 webserver.cc:460] Webserver started at http://127.0.97.62:42089/ using document root <none> and password file <none>
I20260927 05:06:27.400063 5991 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:27.400177 5991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:27.400434 5991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:27.403180 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "c1340c0dd21b43f5bfa3e31685267b71"
format_stamp: "Formatted at 2026-09-27 05:06:27 on dist-test-slave-hgs0"
server_key: "25d1ff09ccd3bb3a621ba52d60ab1f12"
server_key_iv: "e4b15348a933f6d60559fd9449d3a7b6"
server_key_version: "encryptionkey@0"
I20260927 05:06:27.403872 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "c1340c0dd21b43f5bfa3e31685267b71"
format_stamp: "Formatted at 2026-09-27 05:06:27 on dist-test-slave-hgs0"
server_key: "25d1ff09ccd3bb3a621ba52d60ab1f12"
server_key_iv: "e4b15348a933f6d60559fd9449d3a7b6"
server_key_version: "encryptionkey@0"
I20260927 05:06:27.409006 5991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:06:27.412750 6006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:27.414243 5991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:06:27.414425 5991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "c1340c0dd21b43f5bfa3e31685267b71"
format_stamp: "Formatted at 2026-09-27 05:06:27 on dist-test-slave-hgs0"
server_key: "25d1ff09ccd3bb3a621ba52d60ab1f12"
server_key_iv: "e4b15348a933f6d60559fd9449d3a7b6"
server_key_version: "encryptionkey@0"
I20260927 05:06:27.414655 5991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:27.432770 5991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:27.436584 5991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:27.436862 5991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:27.454545 5991 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46627
I20260927 05:06:27.454566 6057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46627 every 8 connection(s)
I20260927 05:06:27.456048 5991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:27.458868 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 5991
I20260927 05:06:27.459075 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:27.459689 388 external_mini_cluster.cc:1428] Setting key 0ffbd523e6f9911048318f074a813538
I20260927 05:06:27.459993 6058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:27.474128 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71: Bootstrap starting.
Sep 27 05:06:27 dist-test-slave-hgs0 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:27.477587 6058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:27.478623 6058 log.cc:824] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:27.481914 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71: No bootstrap required, opened a new log
I20260927 05:06:27.482954 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:27.461805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.0.97.62:46627)
0927 05:06:27.462606 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:27.462623 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:27.462671 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:06:27.464867 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:27.464896 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:27.465435 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:27.465900 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:27.467092 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:27.469071 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:27.470291 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:27.471390 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:27.471584 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:27.475644 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:27.475652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:27.475666 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:27.475706 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:27.478820 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:27.479656 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:27.479664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:27.479672 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:27.479883 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:27.480395 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:27.480399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:27.480401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:27.480891 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:27.481102 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:27.481563 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 05:06:27.481914 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":310,"threads_started":1}
I20260927 05:06:27.490324 6058 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1340c0dd21b43f5bfa3e31685267b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46627 } }
I20260927 05:06:27.490746 6058 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:27.490857 6058 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1340c0dd21b43f5bfa3e31685267b71, State: Initialized, Role: FOLLOWER
I20260927 05:06:27.491389 6058 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1340c0dd21b43f5bfa3e31685267b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46627 } }
I20260927 05:06:27.491616 6058 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:27.491750 6058 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:27.491909 6058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:27.494243 6058 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1340c0dd21b43f5bfa3e31685267b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46627 } }
I20260927 05:06:27.494750 6058 leader_election.cc:302] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1340c0dd21b43f5bfa3e31685267b71; no voters:
I20260927 05:06:27.495306 6058 leader_election.cc:288] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:27.495433 6063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:27.496142 6063 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [term 1 LEADER]: Becoming Leader. State: Replica: c1340c0dd21b43f5bfa3e31685267b71, State: Running, Role: LEADER
I20260927 05:06:27.496667 6063 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1340c0dd21b43f5bfa3e31685267b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46627 } }
I20260927 05:06:27.497419 6058 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:27.499512 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1340c0dd21b43f5bfa3e31685267b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1340c0dd21b43f5bfa3e31685267b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46627 } } }
I20260927 05:06:27.500041 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:27.500366 6065 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1340c0dd21b43f5bfa3e31685267b71. Latest consensus state: current_term: 1 leader_uuid: "c1340c0dd21b43f5bfa3e31685267b71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1340c0dd21b43f5bfa3e31685267b71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46627 } } }
I20260927 05:06:27.500731 6065 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:27.502099 6069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:27.507432 6069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:27.516789 6069 catalog_manager.cc:1347] Generated new cluster ID: b5263ff4e933479380d9d60b684b8aea
I20260927 05:06:27.516916 6069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:27.546375 6069 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:27.547715 6069 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:27.569514 6069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1340c0dd21b43f5bfa3e31685267b71: Generated new TSK 0
I20260927 05:06:27.570614 6069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:06:27.636155 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46627
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:27.962332 6084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:27.962616 6084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:27.962772 6084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:27.970113 6084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:27.970222 6084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:27.970423 6084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:27.980314 6084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:27.981609 6084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:27.982941 6084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:27.995812 6089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:27.995841 6090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:27.996687 6092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:27.999241 6084 server_base.cc:1034] running on GCE node
I20260927 05:06:27.999845 6084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:28.000826 6084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:28.002025 6084 hybrid_clock.cc:648] HybridClock initialized: now 1790485588001967 us; error 53 us; skew 500 ppm
I20260927 05:06:28.002367 6084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:28 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:28.005633 6084 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:06:28.007228 6084 webserver.cc:460] Webserver started at http://127.0.97.1:33883/ using document root <none> and password file <none>
I20260927 05:06:28.008020 6084 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:28.008163 6084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:28.008473 6084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:28.011507 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "1ac699f24cb94ffb926980d4724d5899"
format_stamp: "Formatted at 2026-09-27 05:06:28 on dist-test-slave-hgs0"
server_key: "968bf1db99cb5d599befa4bbff87af1f"
server_key_iv: "43e17cd72a50ac4359ccb57e72c81c23"
server_key_version: "encryptionkey@0"
I20260927 05:06:28.012316 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1ac699f24cb94ffb926980d4724d5899"
format_stamp: "Formatted at 2026-09-27 05:06:28 on dist-test-slave-hgs0"
server_key: "968bf1db99cb5d599befa4bbff87af1f"
server_key_iv: "43e17cd72a50ac4359ccb57e72c81c23"
server_key_version: "encryptionkey@0"
I20260927 05:06:28.017292 6084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 05:06:28.020886 6099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:28.022185 6084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:06:28.022347 6084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "1ac699f24cb94ffb926980d4724d5899"
format_stamp: "Formatted at 2026-09-27 05:06:28 on dist-test-slave-hgs0"
server_key: "968bf1db99cb5d599befa4bbff87af1f"
server_key_iv: "43e17cd72a50ac4359ccb57e72c81c23"
server_key_version: "encryptionkey@0"
I20260927 05:06:28.022545 6084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:28.054183 6084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:28.058324 6084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:28.058609 6084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:28.059564 6084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:28.060945 6084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:28.061048 6084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:28.061199 6084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:28.061267 6084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:28.093603 6084 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37721
I20260927 05:06:28.093650 6211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37721 every 8 connection(s)
I20260927 05:06:28.094882 6084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:28.097576 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 6084
I20260927 05:06:28.097787 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:06:28.098425 388 external_mini_cluster.cc:1428] Setting key bca1dbf1b3e17773b1c58e91d5ad8535
Sep 27 05:06:28 dist-test-slave-hgs0 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:28.115785 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:28.099217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46951 (local address 127.0.97.62:46627)
0927 05:06:28.099656 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:28.099662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:28.099679 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:06:28.101019 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:28.101024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:28.101101 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:28.101219 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:28.102657 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:28.103806 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:28.105426 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:28.106169 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:28.106305 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:28.110331 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:28.110336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:28.110339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:28.110370 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:28.112873 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:28.113600 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:28.113607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:28.113615 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:28.113720 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:28.114129 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:28.114133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:28.114136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:28.114391 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:28.114541 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:28.115211 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 05:06:28.115427 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":229,"threads_started":1}
I20260927 05:06:28.116796 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:28.098105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46627 (local address 127.0.97.1:46951)
0927 05:06:28.099032 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:28.099084 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:28.100757 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:28.101426 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:28.101434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:28.101938 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:28.102481 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:28.102498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:28.104002 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:28.104007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:28.105278 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:28.105289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:28.106404 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:28.106412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:28.106735 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:28.107477 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:28.107515 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:28.110129 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:28.113024 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:28.113033 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:28.113050 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:28.113476 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:28.113825 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:28.113830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:28.113834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:28.114015 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:28.114514 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:28.114522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:28.114847 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 05:06:28.115107 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 05:06:28.115649 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":381,"threads_started":1}
I20260927 05:06:28.118414 6212 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46627
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:06:28.118760 6212 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:28.119596 6212 heartbeater.cc:511] Master 127.0.97.62:46627 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:06:28.121842 6023 ts_manager.cc:194] Registered new tserver with Master: 1ac699f24cb94ffb926980d4724d5899 (127.0.97.1:37721)
I20260927 05:06:28.123235 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:06:28.145300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46627
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:28.465348 6218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:28.465622 6218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:28.465776 6218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:28.472990 6218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:28.473088 6218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:28.473264 6218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:28.482875 6218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:28.484171 6218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:28.485482 6218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:28.499295 6226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:28.499271 6223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:28.499728 6224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:28.500531 6218 server_base.cc:1034] running on GCE node
I20260927 05:06:28.501212 6218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:28.502195 6218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:28.503417 6218 hybrid_clock.cc:648] HybridClock initialized: now 1790485588503339 us; error 75 us; skew 500 ppm
I20260927 05:06:28.503793 6218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:28 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:28.507055 6218 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:06:28.508534 6218 webserver.cc:460] Webserver started at http://127.0.97.2:39281/ using document root <none> and password file <none>
I20260927 05:06:28.509234 6218 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:28.509382 6218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:28.509676 6218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:28.512697 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "5c992fb24b7e47b5ab0d3ec0ff59f8bf"
format_stamp: "Formatted at 2026-09-27 05:06:28 on dist-test-slave-hgs0"
server_key: "3b540d09976ed3561a5bcf21fc0394f2"
server_key_iv: "c1ca9b22a1860d7d97abdf70d7a1aa7f"
server_key_version: "encryptionkey@0"
I20260927 05:06:28.513497 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5c992fb24b7e47b5ab0d3ec0ff59f8bf"
format_stamp: "Formatted at 2026-09-27 05:06:28 on dist-test-slave-hgs0"
server_key: "3b540d09976ed3561a5bcf21fc0394f2"
server_key_iv: "c1ca9b22a1860d7d97abdf70d7a1aa7f"
server_key_version: "encryptionkey@0"
I20260927 05:06:28.518687 6218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:06:28.522210 6233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:28.523574 6218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:06:28.523752 6218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "5c992fb24b7e47b5ab0d3ec0ff59f8bf"
format_stamp: "Formatted at 2026-09-27 05:06:28 on dist-test-slave-hgs0"
server_key: "3b540d09976ed3561a5bcf21fc0394f2"
server_key_iv: "c1ca9b22a1860d7d97abdf70d7a1aa7f"
server_key_version: "encryptionkey@0"
I20260927 05:06:28.523995 6218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:28.559226 6218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:28.563472 6218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:28.563768 6218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:28.564782 6218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:28.566143 6218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:28.566244 6218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:28.566397 6218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:28.566469 6218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:28.599741 6218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39623
I20260927 05:06:28.599843 6345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39623 every 8 connection(s)
I20260927 05:06:28.601115 6218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:28.604296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 6218
I20260927 05:06:28.604477 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:06:28.604902 388 external_mini_cluster.cc:1428] Setting key 117e2723bd44f97c3071e50bd629bed8
Sep 27 05:06:28 dist-test-slave-hgs0 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:28.624560 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:28.603817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53617 (local address 127.0.97.62:46627)
0927 05:06:28.604063 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:28.604070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:28.605396 (+ 1326us) server_negotiation.cc:407] Connection header received
0927 05:06:28.607333 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:28.607337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:28.607402 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:28.607482 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:28.609689 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:28.610394 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:28.613022 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:28.613547 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:28.613684 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:28.617258 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:28.617267 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:28.617275 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:28.617323 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:28.621030 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:28.621827 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:28.621835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:28.621871 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:28.622031 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:28.622515 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:28.622523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:28.622529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:28.622924 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:28.623083 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:28.623798 (+ 715us) server_negotiation.cc:299] Negotiation successful
0927 05:06:28.624116 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 05:06:28.625658 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:28.604358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46627 (local address 127.0.97.2:53617)
0927 05:06:28.605210 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:28.605262 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:28.607037 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:28.607764 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:28.607778 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:28.608620 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:28.609498 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:28.609526 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:28.610557 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:28.610565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:28.612860 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:28.612878 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:28.613831 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:28.613837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:28.614042 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:28.614757 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:28.614784 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:28.616999 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:28.621199 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:28.621207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:28.621223 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:28.621678 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:28.622148 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:28.622152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:28.622154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:28.622346 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:28.623052 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:28.623058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:28.623403 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 05:06:28.623631 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 05:06:28.624139 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":378,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:06:28.628065 6346 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46627
I20260927 05:06:28.628638 6346 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:28.629957 6346 heartbeater.cc:511] Master 127.0.97.62:46627 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:06:28.631879 6023 ts_manager.cc:194] Registered new tserver with Master: 5c992fb24b7e47b5ab0d3ec0ff59f8bf (127.0.97.2:39623)
I20260927 05:06:28.633193 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:06:28.662750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46627
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:28.984011 6351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:28.984342 6351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:28.984510 6351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:28.992045 6351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:28.992175 6351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:28.992359 6351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:29.004209 6351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:29.005435 6351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:29.007146 6351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:29.021515 6356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:29.022744 6359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:29.022744 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:29.023650 6351 server_base.cc:1034] running on GCE node
I20260927 05:06:29.024348 6351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:29.025291 6351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:29.026487 6351 hybrid_clock.cc:648] HybridClock initialized: now 1790485589026434 us; error 44 us; skew 500 ppm
I20260927 05:06:29.026805 6351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:29.030198 6351 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:06:29.031633 6351 webserver.cc:460] Webserver started at http://127.0.97.3:40973/ using document root <none> and password file <none>
I20260927 05:06:29.032415 6351 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:29.032546 6351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:29.032857 6351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:29.036319 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "2f58e311322c419b85b6442b68020daf"
format_stamp: "Formatted at 2026-09-27 05:06:29 on dist-test-slave-hgs0"
server_key: "7082fbce9945d0b8f37ebf8cbac063c9"
server_key_iv: "fc893dec5f1a2b9c1f50461d05da2e22"
server_key_version: "encryptionkey@0"
I20260927 05:06:29.037083 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2f58e311322c419b85b6442b68020daf"
format_stamp: "Formatted at 2026-09-27 05:06:29 on dist-test-slave-hgs0"
server_key: "7082fbce9945d0b8f37ebf8cbac063c9"
server_key_iv: "fc893dec5f1a2b9c1f50461d05da2e22"
server_key_version: "encryptionkey@0"
I20260927 05:06:29.042207 6351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 05:06:29.045626 6366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:29.047066 6351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:06:29.047237 6351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "2f58e311322c419b85b6442b68020daf"
format_stamp: "Formatted at 2026-09-27 05:06:29 on dist-test-slave-hgs0"
server_key: "7082fbce9945d0b8f37ebf8cbac063c9"
server_key_iv: "fc893dec5f1a2b9c1f50461d05da2e22"
server_key_version: "encryptionkey@0"
I20260927 05:06:29.047439 6351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:29.079608 6351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:29.083616 6351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:29.083916 6351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:29.085016 6351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:29.086347 6351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:29.086450 6351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:29.086608 6351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:29.086679 6351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:29.120219 6351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41201
I20260927 05:06:29.120263 6478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41201 every 8 connection(s)
I20260927 05:06:29.121495 6351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:29.124899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 6351
I20260927 05:06:29.125103 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:06:29.125669 388 external_mini_cluster.cc:1428] Setting key 5aa8d1e4b36ffa92d95495a690ea49e3
I20260927 05:06:29.126107 6212 heartbeater.cc:503] Master 127.0.97.62:46627 was elected leader, sending a full tablet report...
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:29.142186 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:29.124058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59589 (local address 127.0.97.62:46627)
0927 05:06:29.124229 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:29.124233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:29.125692 (+ 1459us) server_negotiation.cc:407] Connection header received
0927 05:06:29.128221 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:29.128227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:29.128310 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:29.128440 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:29.129966 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.130728 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.132353 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.132749 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.132885 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:29.136206 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:29.136219 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:29.136223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:29.136255 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:29.138757 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.139561 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.139566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.139570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.139710 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.140265 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.140268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.140271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.140616 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:29.140779 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:29.141597 (+ 818us) server_negotiation.cc:299] Negotiation successful
0927 05:06:29.141837 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:06:29.143290 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:29.124637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46627 (local address 127.0.97.3:59589)
0927 05:06:29.125540 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:29.125577 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:29.127913 (+ 2336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:29.128712 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:29.128726 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:29.129267 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:29.129800 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:29.129816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.130886 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.130891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:29.132198 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:29.132209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.132981 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.132987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:29.133212 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:29.133807 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:29.133832 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:29.135999 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:29.138902 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:29.138913 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:29.138930 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:29.139435 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:29.139842 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:29.139854 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:29.139858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:29.140143 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:29.140750 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:29.140758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:29.141208 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 05:06:29.141489 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 05:06:29.142008 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":383,"threads_started":1}
I20260927 05:06:29.144874 6479 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46627
I20260927 05:06:29.145206 6479 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:29.145995 6479 heartbeater.cc:511] Master 127.0.97.62:46627 requested a full tablet report, sending...
I20260927 05:06:29.147359 6022 ts_manager.cc:194] Registered new tserver with Master: 2f58e311322c419b85b6442b68020daf (127.0.97.3:41201)
I20260927 05:06:29.148260 6022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59589
I20260927 05:06:29.153350 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/test-admin.keytab.
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:29.180775 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 05:06:29.185747 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:29.183320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.0.97.1:37721)
0927 05:06:29.183990 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:29.184000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:29.184034 (+ 34us) server_negotiation.cc:407] Connection header received
0927 05:06:29.184103 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:29.184110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:29.184336 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:29.184502 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:29.185347 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36508: BlockingRecv error: recv got EOF from 127.0.0.1:36508 (error 108)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":271,"threads_started":1}
W20260927 05:06:29.185916 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:29.187484 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:29.189016 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:06:29.204481 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:29.190599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.0.97.1:37721)
0927 05:06:29.190773 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:29.190779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:29.190934 (+ 155us) server_negotiation.cc:407] Connection header received
0927 05:06:29.191077 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:29.191080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:29.191130 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:29.191204 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:29.192249 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.193224 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.194399 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.194925 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.195024 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:29.198736 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:29.198748 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:29.198762 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:29.198792 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:29.201906 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.202495 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.202503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.202505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.202624 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.203012 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.203015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.203017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.203399 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:29.203613 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:29.203905 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 05:06:29.204165 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 05:06:29.210116 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:29.208614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.0.97.1:37721)
0927 05:06:29.208806 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:29.208810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:29.208891 (+ 81us) server_negotiation.cc:407] Connection header received
0927 05:06:29.209012 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:29.209016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:29.209071 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:29.209145 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:29.209829 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36536: BlockingRecv error: recv got EOF from 127.0.0.1:36536 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 05:06:29.210318 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:29.211817 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:29.213150 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:06:29.227437 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:29.214382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.0.97.1:37721)
0927 05:06:29.214570 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:29.214579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:29.214885 (+ 306us) server_negotiation.cc:407] Connection header received
0927 05:06:29.214989 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:29.214992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:29.215040 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:29.215113 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:29.216042 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.216997 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.218221 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.218664 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.218756 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:29.222249 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:29.222254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:29.222257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:29.222286 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:29.224997 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.225612 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.225617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.225619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.225813 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.226141 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.226144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.226146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.226460 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:29.226675 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:29.226909 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 05:06:29.227066 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 05:06:29.231593 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:29.230129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.0.97.1:37721)
0927 05:06:29.230315 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:29.230321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:29.230503 (+ 182us) server_negotiation.cc:407] Connection header received
0927 05:06:29.230619 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:29.230622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:29.230684 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:29.230759 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:29.231371 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36556: BlockingRecv error: recv got EOF from 127.0.0.1:36556 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 05:06:29.231740 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790485535376807-388-0/krb5kdc/krb5cc' of type FILE
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:29.233259 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:29.234618 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:06:29.247805 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:29.235928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.0.97.1:37721)
0927 05:06:29.236171 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:29.236178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:29.236383 (+ 205us) server_negotiation.cc:407] Connection header received
0927 05:06:29.236481 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:29.236484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:29.236535 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:29.236610 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:29.237459 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.238138 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.239394 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.239774 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.239867 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:29.243199 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:29.243204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:29.243207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:29.243236 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:29.245624 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.246211 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.246214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.246217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.246384 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.246751 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.246754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.246756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.246995 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:29.247220 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:29.247417 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 05:06:29.247562 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 05:06:29.249400 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 6084
I20260927 05:06:29.260195 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:29.250098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.0.97.1:37721)
0927 05:06:29.250230 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:29.250234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:29.250623 (+ 389us) server_negotiation.cc:407] Connection header received
0927 05:06:29.250800 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:29.250803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:29.250854 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:29.250929 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:29.251735 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.252476 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.253595 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.253985 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.254071 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:29.255777 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:29.255781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:29.255784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:29.255809 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:29.258064 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.258597 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.258601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.258603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.258715 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.259009 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.259012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.259014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.259223 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:29.259413 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:29.259653 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 05:06:29.259841 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 05:06:29.260728 6207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:29.344499 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 6084
I20260927 05:06:29.360543 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 6218
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:06:29.375200 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:29.361551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.0.97.2:39623)
0927 05:06:29.362125 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:29.362132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:29.362157 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:06:29.362213 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:29.362219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:29.362435 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:29.362566 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:29.363428 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.364582 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.365770 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.366334 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.366437 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:29.369982 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:29.369992 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:29.370007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:29.370044 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:29.372910 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.373512 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.373519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.373521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.373633 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.373937 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.373940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.373942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.374232 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:29.374429 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:29.374730 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 05:06:29.374913 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":317,"threads_started":1}
I20260927 05:06:29.376472 6341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:29.458081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 6218
I20260927 05:06:29.474130 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 6351
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:06:29.488667 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:29.475001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.0.97.3:41201)
0927 05:06:29.475567 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:29.475574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:29.475603 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:06:29.475659 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:29.475665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:29.475891 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:29.476071 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:29.477000 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.478011 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.479166 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:29.479692 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:29.479792 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:29.483502 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:29.483511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:29.483525 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:29.483553 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:29.486299 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.486867 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.486873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.486875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.486979 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:29.487283 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:29.487286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:29.487288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:29.487588 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:29.487833 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:29.488105 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 05:06:29.488384 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":282,"threads_started":1}
I20260927 05:06:29.489956 6474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:29.572553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 6351
I20260927 05:06:29.588546 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 5991
I20260927 05:06:29.589721 6053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:29.691046 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 5991
2026-09-27T05:06:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3957 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[6506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[6506](info): set up 2 sockets
Sep 27 05:06:29 dist-test-slave-hgs0 krb5kdc[6506](info): commencing operation
krb5kdc: starting...
W20260927 05:06:30.780608 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:06:30 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 05:06:30.983259 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46281
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:31.301820 6519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:31.302105 6519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:31.302191 6519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:31.309254 6519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:31.309358 6519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:31.309403 6519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:31.309459 6519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:31.309504 6519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:31.318849 6519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:31.320087 6519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:31.321317 6519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:31.332844 6524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:31.332888 6525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:31.332935 6527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:31.334091 6519 server_base.cc:1034] running on GCE node
I20260927 05:06:31.334757 6519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:31.336052 6519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:31.337265 6519 hybrid_clock.cc:648] HybridClock initialized: now 1790485591337163 us; error 87 us; skew 500 ppm
I20260927 05:06:31.337604 6519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:31 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485591, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:31.347146 6519 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 05:06:31.348786 6519 webserver.cc:460] Webserver started at http://127.0.97.62:35489/ using document root <none> and password file <none>
I20260927 05:06:31.349485 6519 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:31.349632 6519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:31.349941 6519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:31.352887 6519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "3169e443d11645c7b6a8f7c6f7fe2c99"
format_stamp: "Formatted at 2026-09-27 05:06:31 on dist-test-slave-hgs0"
server_key: "dff9ef1280f890e0b3000155cf394dcb"
server_key_iv: "54f0d9c59fccd00b7c5e4f92c808730b"
server_key_version: "encryptionkey@0"
I20260927 05:06:31.353615 6519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "3169e443d11645c7b6a8f7c6f7fe2c99"
format_stamp: "Formatted at 2026-09-27 05:06:31 on dist-test-slave-hgs0"
server_key: "dff9ef1280f890e0b3000155cf394dcb"
server_key_iv: "54f0d9c59fccd00b7c5e4f92c808730b"
server_key_version: "encryptionkey@0"
I20260927 05:06:31.360368 6519 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 05:06:31.364248 6534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:31.365625 6519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:06:31.365798 6519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "3169e443d11645c7b6a8f7c6f7fe2c99"
format_stamp: "Formatted at 2026-09-27 05:06:31 on dist-test-slave-hgs0"
server_key: "dff9ef1280f890e0b3000155cf394dcb"
server_key_iv: "54f0d9c59fccd00b7c5e4f92c808730b"
server_key_version: "encryptionkey@0"
I20260927 05:06:31.365990 6519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:31.421604 6519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:31.425653 6519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:31.425948 6519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:31.444607 6519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46281
I20260927 05:06:31.444655 6585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46281 every 8 connection(s)
I20260927 05:06:31.445972 6519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:31.449599 6586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:31.452314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 6519
I20260927 05:06:31.452513 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:31.453132 388 external_mini_cluster.cc:1428] Setting key f5d3c538aad2baca992a2b7fe51367e1
I20260927 05:06:31.461548 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99: Bootstrap starting.
I20260927 05:06:31.465404 6586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:31.466758 6586 log.cc:824] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:06:31 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:31.470285 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99: No bootstrap required, opened a new log
I20260927 05:06:31.477031 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:31.455102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.0.97.62:46281)
0927 05:06:31.455809 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:31.455820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:31.455861 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:06:31.457522 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:31.457543 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:31.457930 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:31.458237 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:31.459921 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:31.461278 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:31.462967 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:31.463659 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:31.463787 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:31.468646 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:31.468657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:31.468676 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:31.468721 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:31.472745 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:31.473367 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:31.473377 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:31.473390 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:31.473602 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:31.473936 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:31.473941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:31.473945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:31.474627 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:31.474920 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:31.475172 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 05:06:31.475600 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":319,"threads_started":1}
I20260927 05:06:31.480543 6586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } }
I20260927 05:06:31.480851 6586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:31.480928 6586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3169e443d11645c7b6a8f7c6f7fe2c99, State: Initialized, Role: FOLLOWER
I20260927 05:06:31.481385 6586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } }
I20260927 05:06:31.481568 6586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:31.481657 6586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:31.481767 6586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:31.483731 6586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } }
I20260927 05:06:31.484206 6586 leader_election.cc:302] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3169e443d11645c7b6a8f7c6f7fe2c99; no voters:
I20260927 05:06:31.484819 6586 leader_election.cc:288] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:31.484937 6591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:31.485857 6591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 1 LEADER]: Becoming Leader. State: Replica: 3169e443d11645c7b6a8f7c6f7fe2c99, State: Running, Role: LEADER
I20260927 05:06:31.486342 6591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } }
I20260927 05:06:31.486800 6586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:31.489107 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } }
I20260927 05:06:31.489418 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:31.489863 6593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3169e443d11645c7b6a8f7c6f7fe2c99. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } }
I20260927 05:06:31.490113 6593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:31.493981 6596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:31.499107 6596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:31.508819 6596 catalog_manager.cc:1347] Generated new cluster ID: c5b3015a0d3e49ebbdc03f7c087ab107
I20260927 05:06:31.508946 6596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:31.523290 6596 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:31.524670 6596 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:31.535486 6596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99: Generated new TSK 0
I20260927 05:06:31.536377 6596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 05:06:31.601102 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46281
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:31.947199 6612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:31.947484 6612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:31.947584 6612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:31.955046 6612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:31.955156 6612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:31.955353 6612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:31.965163 6612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:31.966485 6612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:31.967979 6612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:31.981923 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:31.981885 6620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:31.982059 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:31.982829 6612 server_base.cc:1034] running on GCE node
I20260927 05:06:31.983440 6612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:31.984424 6612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:31.985652 6612 hybrid_clock.cc:648] HybridClock initialized: now 1790485591985611 us; error 57 us; skew 500 ppm
I20260927 05:06:31.985963 6612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:31 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485591, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:31.989179 6612 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 05:06:31.990612 6612 webserver.cc:460] Webserver started at http://127.0.97.1:39013/ using document root <none> and password file <none>
I20260927 05:06:31.991281 6612 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:31.991428 6612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:31.991724 6612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:31.994747 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "ec4f8eee7df847efb91934b3963549e8"
format_stamp: "Formatted at 2026-09-27 05:06:31 on dist-test-slave-hgs0"
server_key: "eefd2f07402f88464fca5e0978c2cd78"
server_key_iv: "0a7e9742ca08c2d37e89260e4b20235d"
server_key_version: "encryptionkey@0"
I20260927 05:06:31.995515 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ec4f8eee7df847efb91934b3963549e8"
format_stamp: "Formatted at 2026-09-27 05:06:31 on dist-test-slave-hgs0"
server_key: "eefd2f07402f88464fca5e0978c2cd78"
server_key_iv: "0a7e9742ca08c2d37e89260e4b20235d"
server_key_version: "encryptionkey@0"
I20260927 05:06:32.000603 6612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:06:32.004141 6627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:32.005455 6612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:06:32.005645 6612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "ec4f8eee7df847efb91934b3963549e8"
format_stamp: "Formatted at 2026-09-27 05:06:31 on dist-test-slave-hgs0"
server_key: "eefd2f07402f88464fca5e0978c2cd78"
server_key_iv: "0a7e9742ca08c2d37e89260e4b20235d"
server_key_version: "encryptionkey@0"
I20260927 05:06:32.005844 6612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:32.036754 6612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:32.041280 6612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:32.041604 6612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:32.042558 6612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:32.043871 6612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:32.043994 6612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:32.044143 6612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:32.044229 6612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:32.075510 6612 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45703
I20260927 05:06:32.075554 6739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45703 every 8 connection(s)
I20260927 05:06:32.076906 6612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:32.084019 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 6612
I20260927 05:06:32.084209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:06:32.084824 388 external_mini_cluster.cc:1428] Setting key c4d7052d6a05a26c65e0742352e8e752
Sep 27 05:06:32 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485591, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:32.098567 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:32.079507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53981 (local address 127.0.97.62:46281)
0927 05:06:32.079985 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:32.079994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:32.080853 (+ 859us) server_negotiation.cc:407] Connection header received
0927 05:06:32.082650 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:32.082656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:32.082757 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:32.082874 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:32.084455 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:32.085258 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:32.087149 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:32.087535 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:32.087670 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:32.091704 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:32.091709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:32.091712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:32.091743 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:32.095220 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:32.096092 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:32.096100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:32.096106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:32.096253 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:32.096661 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:32.096669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:32.096675 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:32.097027 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:32.097171 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:32.097848 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 05:06:32.098090 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":235,"threads_started":1}
I20260927 05:06:32.099270 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:32.080026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.1:53981)
0927 05:06:32.080696 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:32.080738 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:32.082378 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:32.083113 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:32.083121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:32.083596 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:32.084284 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:32.084309 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:32.085406 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:32.085412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:32.087001 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:32.087012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:32.087767 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:32.087774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:32.088087 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:32.088754 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:32.088776 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:32.091489 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:32.095371 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:32.095378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:32.095391 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:32.095957 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:32.096368 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:32.096371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:32.096373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:32.096548 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:32.097140 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:32.097145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:32.097466 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 05:06:32.097682 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 05:06:32.098084 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":270,"threads_started":1}
I20260927 05:06:32.100896 6740 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46281
I20260927 05:06:32.101233 6740 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:32.102128 6740 heartbeater.cc:511] Master 127.0.97.62:46281 requested a full tablet report, sending...
I20260927 05:06:32.104517 6551 ts_manager.cc:194] Registered new tserver with Master: ec4f8eee7df847efb91934b3963549e8 (127.0.97.1:45703)
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:06:32.105965 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 05:06:32.133533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46281
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:32.452586 6746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:32.452911 6746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:32.453082 6746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:32.460237 6746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:32.460407 6746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:32.460602 6746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:32.470418 6746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:32.471850 6746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:32.473137 6746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:32.486322 6751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:32.486358 6752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:32.486413 6754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:32.487128 6746 server_base.cc:1034] running on GCE node
I20260927 05:06:32.487855 6746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:32.488795 6746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:32.489998 6746 hybrid_clock.cc:648] HybridClock initialized: now 1790485592489961 us; error 30 us; skew 500 ppm
I20260927 05:06:32.490309 6746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:32 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485592, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:32.493498 6746 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 05:06:32.494951 6746 webserver.cc:460] Webserver started at http://127.0.97.2:43631/ using document root <none> and password file <none>
I20260927 05:06:32.495572 6746 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:32.495708 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:32.496047 6746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:32.498790 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "1691dc7882c1481eb19f193d542bb678"
format_stamp: "Formatted at 2026-09-27 05:06:32 on dist-test-slave-hgs0"
server_key: "8d29f0c993d73e4295cf16d6c286796d"
server_key_iv: "e64fe76a16dbe8423028b9e289bb2d1c"
server_key_version: "encryptionkey@0"
I20260927 05:06:32.499513 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1691dc7882c1481eb19f193d542bb678"
format_stamp: "Formatted at 2026-09-27 05:06:32 on dist-test-slave-hgs0"
server_key: "8d29f0c993d73e4295cf16d6c286796d"
server_key_iv: "e64fe76a16dbe8423028b9e289bb2d1c"
server_key_version: "encryptionkey@0"
I20260927 05:06:32.504859 6746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 05:06:32.508191 6761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:32.509516 6746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:06:32.509682 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "1691dc7882c1481eb19f193d542bb678"
format_stamp: "Formatted at 2026-09-27 05:06:32 on dist-test-slave-hgs0"
server_key: "8d29f0c993d73e4295cf16d6c286796d"
server_key_iv: "e64fe76a16dbe8423028b9e289bb2d1c"
server_key_version: "encryptionkey@0"
I20260927 05:06:32.509894 6746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:32.520416 6746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:32.524655 6746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:32.524945 6746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:32.525846 6746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:32.527105 6746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:32.527204 6746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:32.527354 6746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:32.527426 6746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:32.559567 6746 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33669
I20260927 05:06:32.559602 6873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33669 every 8 connection(s)
I20260927 05:06:32.560845 6746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:32.563436 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 6746
I20260927 05:06:32.563606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:06:32.564095 388 external_mini_cluster.cc:1428] Setting key a703dae3b9fd1468bfe53cfce8ac5347
Sep 27 05:06:32 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485592, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:32.581467 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:32.563426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55731 (local address 127.0.97.62:46281)
0927 05:06:32.563647 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:32.563655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:32.565620 (+ 1965us) server_negotiation.cc:407] Connection header received
0927 05:06:32.566974 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:32.566982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:32.567077 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:32.567190 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:32.568805 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:32.569954 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:32.571469 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:32.572057 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:32.572200 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:32.575498 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:32.575503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:32.575506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:32.575537 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:32.578051 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:32.578736 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:32.578742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:32.578746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:32.578885 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:32.579315 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:32.579322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:32.579327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:32.579641 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:32.579785 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:32.580731 (+ 946us) server_negotiation.cc:299] Negotiation successful
0927 05:06:32.580996 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:06:32.582073 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:32.564023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.2:55731)
0927 05:06:32.564855 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:32.564900 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:32.566704 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:32.567375 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:32.567383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:32.567987 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:32.568633 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:32.568652 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:32.570125 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:32.570129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:32.571307 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:32.571323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:32.572282 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:32.572289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:32.572513 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:32.573189 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:32.573210 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:32.575256 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:32.578198 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:32.578206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:32.578223 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:32.578616 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:32.578997 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:32.579002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:32.579006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:32.579204 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:32.579800 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:32.579811 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:32.580307 (+ 496us) client_negotiation.cc:769] Sending connection context
0927 05:06:32.580592 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 05:06:32.580975 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":372,"threads_started":1}
I20260927 05:06:32.583572 6874 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46281
I20260927 05:06:32.583921 6874 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:06:32.584761 6874 heartbeater.cc:511] Master 127.0.97.62:46281 requested a full tablet report, sending...
I20260927 05:06:32.586130 6551 ts_manager.cc:194] Registered new tserver with Master: 1691dc7882c1481eb19f193d542bb678 (127.0.97.2:33669)
I20260927 05:06:32.587038 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 05:06:32.611634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46281
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:32.932961 6879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:32.933233 6879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:32.933385 6879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:32.940490 6879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:32.940618 6879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:32.940797 6879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:32.950263 6879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:32.951537 6879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:32.953212 6879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:32.965716 6884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:32.965757 6885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:32.965669 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:32.966715 6879 server_base.cc:1034] running on GCE node
I20260927 05:06:32.967350 6879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:32.968328 6879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:32.969523 6879 hybrid_clock.cc:648] HybridClock initialized: now 1790485592969457 us; error 54 us; skew 500 ppm
I20260927 05:06:32.969861 6879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:32 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485592, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:32.973035 6879 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 05:06:32.974361 6879 webserver.cc:460] Webserver started at http://127.0.97.3:42043/ using document root <none> and password file <none>
I20260927 05:06:32.975036 6879 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:32.975157 6879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:32.975443 6879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:32.978387 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "4d5a2a91d9a5430a9138d357ee7c1cff"
format_stamp: "Formatted at 2026-09-27 05:06:32 on dist-test-slave-hgs0"
server_key: "4046db49c57c6d64d0cd7634e313fc94"
server_key_iv: "9c8c640e8d89d058e62b48999cc7bd77"
server_key_version: "encryptionkey@0"
I20260927 05:06:32.979139 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4d5a2a91d9a5430a9138d357ee7c1cff"
format_stamp: "Formatted at 2026-09-27 05:06:32 on dist-test-slave-hgs0"
server_key: "4046db49c57c6d64d0cd7634e313fc94"
server_key_iv: "9c8c640e8d89d058e62b48999cc7bd77"
server_key_version: "encryptionkey@0"
I20260927 05:06:32.984187 6879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 05:06:32.987469 6894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:32.988967 6879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:32.989140 6879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "4d5a2a91d9a5430a9138d357ee7c1cff"
format_stamp: "Formatted at 2026-09-27 05:06:32 on dist-test-slave-hgs0"
server_key: "4046db49c57c6d64d0cd7634e313fc94"
server_key_iv: "9c8c640e8d89d058e62b48999cc7bd77"
server_key_version: "encryptionkey@0"
I20260927 05:06:32.989347 6879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:33.018715 6879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:33.023226 6879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:33.023547 6879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:33.024591 6879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:33.025920 6879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:33.025992 6879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:33.026154 6879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:33.026203 6879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:33.059355 6879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40097
I20260927 05:06:33.059408 7006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40097 every 8 connection(s)
I20260927 05:06:33.060655 6879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:33.061060 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 6879
I20260927 05:06:33.061232 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:06:33.061810 388 external_mini_cluster.cc:1428] Setting key 6a6cf163ef56474efae75c1ec939d6be
Sep 27 05:06:33 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485592, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:33.081641 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.063474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41381 (local address 127.0.97.62:46281)
0927 05:06:33.063663 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:33.063667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:33.065132 (+ 1465us) server_negotiation.cc:407] Connection header received
0927 05:06:33.066894 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:33.066899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:33.066979 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:33.067093 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:33.068609 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.069439 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.071151 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.071557 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.071699 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.075163 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:33.075168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:33.075171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:33.075204 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:33.078746 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.079512 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.079518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.079522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.079661 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.080075 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.080079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.080083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.080424 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:33.080562 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:33.081208 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 05:06:33.081414 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:06:33.082672 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.064166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.3:41381)
0927 05:06:33.064989 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:33.065027 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:33.066635 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:33.067317 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:33.067325 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:33.067807 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:33.068440 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.068457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.069606 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.069610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.071002 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.071018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.071783 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.071792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.072080 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.072806 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:33.072827 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:33.074952 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:33.078918 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.078926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.078942 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.079383 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.079759 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.079763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.079765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.079928 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.080538 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:33.080543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:33.080828 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 05:06:33.081048 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 05:06:33.081560 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":309,"threads_started":1}
I20260927 05:06:33.084038 7007 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46281
I20260927 05:06:33.084331 7007 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:33.085009 7007 heartbeater.cc:511] Master 127.0.97.62:46281 requested a full tablet report, sending...
I20260927 05:06:33.086336 6551 ts_manager.cc:194] Registered new tserver with Master: 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097)
I20260927 05:06:33.087205 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41381
I20260927 05:06:33.089483 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260927 05:06:33.109016 6740 heartbeater.cc:503] Master 127.0.97.62:46281 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260927 05:06:33.135706 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38783
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46281,127.0.97.61:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:33.446378 7012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:33.446655 7012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:33.446813 7012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:33.453922 7012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:33.454051 7012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:33.454159 7012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:33.454250 7012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:33.454334 7012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:33.463958 7012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46281,127.0.97.61:38783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
W20260927 05:06:33.464334 7012 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:46281,127.0.97.61:38783'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 05:06:33.465404 7012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:33.466742 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:33.478412 7017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:33.478385 7020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:33.478351 7018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:33.479480 7012 server_base.cc:1034] running on GCE node
I20260927 05:06:33.480300 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:33.481640 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:33.482884 7012 hybrid_clock.cc:648] HybridClock initialized: now 1790485593482810 us; error 55 us; skew 500 ppm
I20260927 05:06:33.483212 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:33 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485593, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:33.486359 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 05:06:33.487921 7012 webserver.cc:460] Webserver started at http://127.0.97.61:33877/ using document root <none> and password file <none>
I20260927 05:06:33.488703 7012 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:33.488786 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:33.489043 7012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:33.491880 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data/instance:
uuid: "cd7692436bd94dcc91d8823bb2692aba"
format_stamp: "Formatted at 2026-09-27 05:06:33 on dist-test-slave-hgs0"
server_key: "e314e1a9e4d751c2845df07f2eea5297"
server_key_iv: "f36767235cecd9a5b7e4d23c2fabfdb5"
server_key_version: "encryptionkey@0"
I20260927 05:06:33.492879 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal/instance:
uuid: "cd7692436bd94dcc91d8823bb2692aba"
format_stamp: "Formatted at 2026-09-27 05:06:33 on dist-test-slave-hgs0"
server_key: "e314e1a9e4d751c2845df07f2eea5297"
server_key_iv: "f36767235cecd9a5b7e4d23c2fabfdb5"
server_key_version: "encryptionkey@0"
I20260927 05:06:33.498107 7012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:06:33.501466 7027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:33.502931 7012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:06:33.503094 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal
uuid: "cd7692436bd94dcc91d8823bb2692aba"
format_stamp: "Formatted at 2026-09-27 05:06:33 on dist-test-slave-hgs0"
server_key: "e314e1a9e4d751c2845df07f2eea5297"
server_key_iv: "f36767235cecd9a5b7e4d23c2fabfdb5"
server_key_version: "encryptionkey@0"
I20260927 05:06:33.503307 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:33.514446 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:33.518373 7012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:33.518649 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:33.537350 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38783
I20260927 05:06:33.537369 7078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38783 every 8 connection(s)
I20260927 05:06:33.538703 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
I20260927 05:06:33.542433 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:33.544520 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 7012
I20260927 05:06:33.551345 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:06:33 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 05:06:33 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485593, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:33.567853 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.546782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.0.97.61:38783)
0927 05:06:33.547545 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:33.547557 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:33.547598 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:06:33.549144 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:33.549164 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:33.549550 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:33.549944 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:33.551302 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.552670 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.554011 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.555023 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.555203 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.559780 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:33.559793 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:33.559812 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:33.559862 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:33.563666 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.564515 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.564524 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.564532 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.564681 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.565081 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.565087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.565090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.565699 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:33.565986 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:33.566427 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 05:06:33.566919 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":325,"threads_started":1}
I20260927 05:06:33.569276 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.553825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.0.97.62:46281)
0927 05:06:33.554049 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:33.554055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:33.554522 (+ 467us) server_negotiation.cc:407] Connection header received
0927 05:06:33.554702 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:33.554705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:33.554767 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:33.554863 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:33.556102 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.557274 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.558546 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.559162 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.559299 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.563460 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:33.563467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:33.563472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:33.563513 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:33.566471 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.567204 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.567209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.567213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.567333 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.567698 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.567705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.567708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.568073 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:33.568207 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:33.568709 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 05:06:33.568989 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 05:06:33.569587 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.553697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:50300)
0927 05:06:33.554384 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:33.554426 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:33.554577 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:33.554939 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:33.554946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:33.555685 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:33.555902 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.555910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.557417 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.557420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.558398 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.558407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.559393 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.559400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.559545 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.560274 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:33.560295 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:33.563254 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:33.566602 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.566614 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.566620 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.567082 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.567445 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.567454 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.567458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.567594 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.568204 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:33.568211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:33.568390 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 05:06:33.568612 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 05:06:33.568902 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":295,"threads_started":1}
I20260927 05:06:33.571346 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:06:33 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485593, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 05:06:33.582690 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.572118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38783 (local address 127.0.0.1:44946)
0927 05:06:33.572264 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:33.572278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:33.572370 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:33.572640 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:33.572643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:33.572962 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:33.573088 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.573095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.574061 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.574065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.574916 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.574926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.575523 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.575527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.575662 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.576180 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:33.576195 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:33.578008 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:33.580793 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.580796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.580799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.581136 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.581441 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.581444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.581446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.581557 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.582003 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:33.582006 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:33.582119 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 05:06:33.582221 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 05:06:33.582430 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 05:06:33.582775 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.572072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.0.97.61:38783)
0927 05:06:33.572231 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:33.572240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:33.572347 (+ 107us) server_negotiation.cc:407] Connection header received
0927 05:06:33.572440 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:33.572444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:33.572504 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:33.572576 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:33.573200 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.573955 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.575026 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.575406 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.575503 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.578164 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:33.578169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:33.578172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:33.578197 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:33.580679 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.581235 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.581238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.581241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.581339 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.581642 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.581645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.581647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.581909 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:33.582014 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:33.582298 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 05:06:33.582545 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":49}
I20260927 05:06:33.587512 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Bootstrap starting.
I20260927 05:06:33.589772 6874 heartbeater.cc:503] Master 127.0.97.62:46281 was elected leader, sending a full tablet report...
I20260927 05:06:33.590832 7079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:33.592036 7079 log.cc:824] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:33.594828 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: No bootstrap required, opened a new log
I20260927 05:06:33.601696 7079 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } }
I20260927 05:06:33.602001 7079 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:33.602061 7079 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7692436bd94dcc91d8823bb2692aba, State: Initialized, Role: FOLLOWER
I20260927 05:06:33.602623 7079 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } }
I20260927 05:06:33.603364 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } } }
I20260927 05:06:33.603691 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:06:33.604458 7079 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: configured and running, proceeding with master startup.
W20260927 05:06:33.613379 7096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:06:33.613526 7096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:06:33.615447 7012 master_runner.cc:428] Detected that this master cd7692436bd94dcc91d8823bb2692aba is joining an existing cluster
I20260927 05:06:33.615540 7012 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:38783
I20260927 05:06:33.638404 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.619424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.0.97.62:46281)
0927 05:06:33.619614 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:33.619621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:33.620487 (+ 866us) server_negotiation.cc:407] Connection header received
0927 05:06:33.620801 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:33.620805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:33.620870 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:33.620968 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:33.624795 (+ 3827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.625929 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.628266 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.628887 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.628999 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.631688 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:33.631695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:33.631700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:33.631738 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:33.635233 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.636078 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.636084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.636091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.636228 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.636685 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.636690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.636693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.637101 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:33.637234 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:33.637800 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 05:06:33.638035 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:06:33.638736 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.619537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.0.97.61:38783)
0927 05:06:33.619719 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:33.619723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:33.620937 (+ 1214us) server_negotiation.cc:407] Connection header received
0927 05:06:33.621096 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:33.621103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:33.621174 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:33.621248 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:33.624769 (+ 3521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.625925 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.628249 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.629164 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.629255 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.632230 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:33.632237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:33.632243 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:33.632288 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:33.635855 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.636682 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.636687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.636690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.636843 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.637226 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.637230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.637233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.637581 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:33.637731 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:33.638175 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 05:06:33.638397 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:06:33.639786 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.620068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38783 (local address 127.0.0.1:44958)
0927 05:06:33.620825 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:33.620845 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:33.620981 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:33.621539 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:33.621544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:33.622262 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:33.624627 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.624636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.626138 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.626143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.628099 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.628114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.629551 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.629556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.629741 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.630387 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:33.630413 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:33.632071 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:33.636019 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.636025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.636030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.636583 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.636975 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.636978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.636981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.637127 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.637792 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:33.637796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:33.637938 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 05:06:33.638979 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0927 05:06:33.639275 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":155,"thread_start_us":373,"threads_started":1}
I20260927 05:06:33.639827 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.619559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:50308)
0927 05:06:33.620345 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:33.620375 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:33.620614 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:33.621190 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:33.621195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:33.622374 (+ 1179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:33.624616 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.624629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.626138 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.626143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.628102 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.628115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.629295 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.629301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.629498 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.630161 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:33.630209 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:33.631463 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:33.635402 (+ 3939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.635407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.635411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.635924 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.636379 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.636383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.636388 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.636556 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.637312 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:33.637319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:33.637514 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 05:06:33.639128 (+ 1614us) client_negotiation.cc:240] Negotiation successful
0927 05:06:33.639405 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":200,"thread_start_us":473,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 27 05:06:33 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485591, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 05:06:33.662528 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.646785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.0.97.61:38783)
0927 05:06:33.646940 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:33.646946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:33.647117 (+ 171us) server_negotiation.cc:407] Connection header received
0927 05:06:33.647462 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:33.647467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:33.647554 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:33.647645 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:33.648871 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.649675 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.651318 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.651798 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.651977 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.655057 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:33.655062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:33.655065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:33.655093 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:33.659656 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.660483 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.660497 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.660499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.660650 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.661057 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.661061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.661063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.661320 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:33.661490 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:33.661931 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 05:06:33.662159 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:06:33.662580 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.646417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38783 (local address 127.0.0.1:44960)
0927 05:06:33.647003 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:33.647047 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:33.647244 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:33.647885 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:33.647890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:33.648506 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:33.648695 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.648703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.649860 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.649865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.651186 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.651197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.651988 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.651994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.652181 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.652796 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:33.652813 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:33.654867 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:33.659815 (+ 4948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.659831 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.659840 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.660343 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.660769 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.660774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.660778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.660927 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.661467 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:33.661476 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:33.661651 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 05:06:33.661831 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 05:06:33.662164 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":540,"threads_started":1}
I20260927 05:06:33.664778 6551 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:38783: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 38783
}
attrs {
promote: true
}
}
dest_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99"
cas_config_opid_index: -1
I20260927 05:06:33.665764 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } }
I20260927 05:06:33.674149 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cd7692436bd94dcc91d8823bb2692aba (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } } }
I20260927 05:06:33.675869 7054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:33.678170 7108 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:38783
I20260927 05:06:33.678683 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } } }
I20260927 05:06:33.678987 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:33.679050 7012 master.cc:537] Master@127.0.97.61:38783 shutting down...
I20260927 05:06:33.680877 7054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Refusing update from remote peer 3169e443d11645c7b6a8f7c6f7fe2c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 05:06:33.682276 7109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260927 05:06:33.685276 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44619
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46281,127.0.97.61:38783,127.0.97.60:44619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:33.686405 6538 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 05:06:33.693428 7012 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:33.693939 7012 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:33.694149 7012 tablet_replica.cc:330] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: stopping tablet replica
W20260927 05:06:33.694121 6538 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 -> Peer cd7692436bd94dcc91d8823bb2692aba (127.0.97.61:38783): Couldn't send request to peer cd7692436bd94dcc91d8823bb2692aba. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 05:06:33.701085 7012 master.cc:552] Master@127.0.97.61:38783 shutdown complete.
I20260927 05:06:33.701188 7012 master_runner.cc:305] Clearing existing system tablet
I20260927 05:06:33.701565 7012 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:33.701673 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:33.703843 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 05:06:33.708453 7114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:33.708818 7012 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:33.709167 7012 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260927 05:06:33.709264 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal
uuid: "cd7692436bd94dcc91d8823bb2692aba"
format_stamp: "Formatted at 2026-09-27 05:06:33 on dist-test-slave-hgs0"
server_key: "e314e1a9e4d751c2845df07f2eea5297"
server_key_iv: "f36767235cecd9a5b7e4d23c2fabfdb5"
server_key_version: "encryptionkey@0"
I20260927 05:06:33.712859 7012 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:33.718201 7012 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 05:06:33.718382 7012 log.cc:1196] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 05:06:33.718770 7012 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Deleting consensus metadata
I20260927 05:06:33.719033 7012 master_runner.cc:315] Copying system tablet from 127.0.97.62:46281
I20260927 05:06:33.725521 7012 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46281
I20260927 05:06:33.739590 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.726446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.97.62:46281)
0927 05:06:33.726720 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:33.726730 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:33.728433 (+ 1703us) server_negotiation.cc:407] Connection header received
0927 05:06:33.728649 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:33.728655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:33.728741 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:33.728850 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:33.729675 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.730387 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.731830 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.732439 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.732569 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.734292 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:33.734298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:33.734303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:33.734344 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:33.737018 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.737643 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.737649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.737653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.737783 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:33.738142 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:33.738150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:33.738153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:33.738432 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:33.738583 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:33.738990 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 05:06:33.739206 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:06:33.739670 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:33.726295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:50320)
0927 05:06:33.728304 (+ 2009us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:33.728335 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:33.728510 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:33.728891 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:33.728898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:33.729304 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:33.729524 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.729532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.730531 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.730537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.731690 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:33.731702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:33.732582 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:33.732588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:33.732752 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:33.733394 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:33.733434 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:33.734102 (+ 668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:33.737157 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.737162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.737166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.737524 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.737892 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:33.737900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:33.737904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:33.738034 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:33.738553 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:33.738558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:33.738746 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 05:06:33.738900 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 05:06:33.739208 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1889,"thread_start_us":1813,"threads_started":1}
I20260927 05:06:33.740782 6571 tablet_copy_service.cc:140] P 3169e443d11645c7b6a8f7c6f7fe2c99: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cd7692436bd94dcc91d8823bb2692aba ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:50320)
I20260927 05:06:33.741034 6571 tablet_copy_service.cc:161] P 3169e443d11645c7b6a8f7c6f7fe2c99: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cd7692436bd94dcc91d8823bb2692aba at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:50320: session id = cd7692436bd94dcc91d8823bb2692aba-00000000000000000000000000000000
I20260927 05:06:33.744099 6571 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 05:06:33.745981 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 05:06:33.750514 7012 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: tablet copy: Starting download of 0 data blocks...
I20260927 05:06:33.750797 7012 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: tablet copy: Starting download of 1 WAL segments...
I20260927 05:06:33.753129 7012 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 05:06:33.756155 6571 tablet_copy_service.cc:342] P 3169e443d11645c7b6a8f7c6f7fe2c99: Request end of tablet copy session cd7692436bd94dcc91d8823bb2692aba-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:50320
I20260927 05:06:33.756333 6571 tablet_copy_service.cc:433] P 3169e443d11645c7b6a8f7c6f7fe2c99: ending tablet copy session cd7692436bd94dcc91d8823bb2692aba-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cd7692436bd94dcc91d8823bb2692aba
I20260927 05:06:33.766435 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:33.774084 7121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:33.779510 7122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:33.779529 7124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:33.780230 7012 server_base.cc:1034] running on GCE node
I20260927 05:06:33.780577 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:33.781211 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:33.782490 7012 hybrid_clock.cc:648] HybridClock initialized: now 1790485593782425 us; error 32 us; skew 500 ppm
I20260927 05:06:33.782709 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:33 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485593, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:33.784597 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260927 05:06:33.785815 7012 webserver.cc:460] Webserver started at http://127.0.97.61:38973/ using document root <none> and password file <none>
I20260927 05:06:33.786132 7012 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:33.786219 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:33.788746 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:06:33.790419 7131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:33.790823 7012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260927 05:06:33.790977 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal
uuid: "cd7692436bd94dcc91d8823bb2692aba"
format_stamp: "Formatted at 2026-09-27 05:06:33 on dist-test-slave-hgs0"
server_key: "e314e1a9e4d751c2845df07f2eea5297"
server_key_iv: "f36767235cecd9a5b7e4d23c2fabfdb5"
server_key_version: "encryptionkey@0"
I20260927 05:06:33.791138 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:33.833209 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:33.836709 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:33.861457 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38783
I20260927 05:06:33.861506 7182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38783 every 8 connection(s)
I20260927 05:06:33.862363 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
I20260927 05:06:33.869216 7183 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:06:33.875034 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Bootstrap starting.
I20260927 05:06:33.891187 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:06:33.891799 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Bootstrap complete.
I20260927 05:06:33.892922 7183 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } }
I20260927 05:06:33.893172 7183 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd7692436bd94dcc91d8823bb2692aba, State: Initialized, Role: LEARNER
I20260927 05:06:33.893436 7183 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } }
I20260927 05:06:33.894249 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } } }
I20260927 05:06:33.894658 7183 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:33.894658 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: This master's current role is: LEARNER
I20260927 05:06:33.915307 7197 catalog_manager.cc:1259] Loaded cluster ID: c5b3015a0d3e49ebbdc03f7c087ab107
I20260927 05:06:33.915434 7197 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: loading cluster ID for follower catalog manager: success
I20260927 05:06:33.917860 7197 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: acquiring CA information for follower catalog manager: success
I20260927 05:06:33.919780 7197 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:06:34.089723 7007 heartbeater.cc:503] Master 127.0.97.62:46281 was elected leader, sending a full tablet report...
W20260927 05:06:34.107204 7112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:34.107496 7112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:34.107640 7112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:34.118139 7112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:34.118289 7112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:34.118413 7112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:34.118513 7112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:34.118578 7112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:34.121805 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.111427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38783 (local address 127.0.0.1:44966)
0927 05:06:34.111673 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.111695 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:34.111835 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:34.114421 (+ 2586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:34.114858 (+ 437us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:34.114894 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:34.115118 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.115132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.117119 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.117125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.119158 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.119175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.120976 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.120982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.121196 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.121259 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 05:06:34.121355 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 05:06:34.121531 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 05:06:34.121909 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.112293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.0.97.61:38783)
0927 05:06:34.114007 (+ 1714us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.114013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.114034 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:06:34.114107 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.114113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.114197 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.114328 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:34.115295 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.116960 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.119326 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.120780 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.120948 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.121205 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.121409 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.121548 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1577,"thread_start_us":1408,"threads_started":1}
I20260927 05:06:34.123414 7158 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 05:06:34.124452 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3169e443d11645c7b6a8f7c6f7fe2c99. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } } }
I20260927 05:06:34.124693 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: This master's current role is: LEARNER
I20260927 05:06:34.130543 7112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46281,127.0.97.61:38783,127.0.97.60:44619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:34.132509 7112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:34.134058 7112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:34.146318 7203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:34.146348 7204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:34.146384 7206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:34.147193 7112 server_base.cc:1034] running on GCE node
I20260927 05:06:34.148088 7112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:34.149518 7112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:34.150750 7112 hybrid_clock.cc:648] HybridClock initialized: now 1790485594150705 us; error 41 us; skew 500 ppm
I20260927 05:06:34.151163 7112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:34 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485594, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:34.154304 7112 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 05:06:34.155727 7112 webserver.cc:460] Webserver started at http://127.0.97.60:35283/ using document root <none> and password file <none>
I20260927 05:06:34.156499 7112 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:34.156647 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:34.156956 7112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:34.159808 7112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data/instance:
uuid: "182fdc6058254c2987252532c20577d8"
format_stamp: "Formatted at 2026-09-27 05:06:34 on dist-test-slave-hgs0"
server_key: "b09846fc61dc5bd163e93f647d482d18"
server_key_iv: "c1fc84e9a8200c47e249cf8d427cf176"
server_key_version: "encryptionkey@0"
I20260927 05:06:34.160669 7112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal/instance:
uuid: "182fdc6058254c2987252532c20577d8"
format_stamp: "Formatted at 2026-09-27 05:06:34 on dist-test-slave-hgs0"
server_key: "b09846fc61dc5bd163e93f647d482d18"
server_key_iv: "c1fc84e9a8200c47e249cf8d427cf176"
server_key_version: "encryptionkey@0"
I20260927 05:06:34.165673 7112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:06:34.169610 7213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:34.170989 7112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 05:06:34.171172 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal
uuid: "182fdc6058254c2987252532c20577d8"
format_stamp: "Formatted at 2026-09-27 05:06:34 on dist-test-slave-hgs0"
server_key: "b09846fc61dc5bd163e93f647d482d18"
server_key_iv: "c1fc84e9a8200c47e249cf8d427cf176"
server_key_version: "encryptionkey@0"
I20260927 05:06:34.171375 7112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:34.184720 7112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:34.188820 7112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:34.189105 7112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:34.209266 7112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44619
I20260927 05:06:34.209309 7264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44619 every 8 connection(s)
I20260927 05:06:34.210640 7112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
I20260927 05:06:34.214589 7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:34.214596 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 7112
I20260927 05:06:34.222810 7265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:06:34 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 05:06:34 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485594, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:34.238483 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.216745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.0.97.60:44619)
0927 05:06:34.217543 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.217554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.217599 (+ 45us) server_negotiation.cc:407] Connection header received
0927 05:06:34.219111 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.219133 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.219515 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.219917 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:34.221791 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.223075 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.224447 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.225406 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.225611 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.230626 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:34.230634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:34.230644 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:34.230680 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:34.234493 (+ 3813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.235154 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.235164 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.235171 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.235306 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.235699 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.235709 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.235711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.236274 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:34.236556 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.237023 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.237416 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":381,"threads_started":1}
I20260927 05:06:34.240331 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.225548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.0.97.62:46281)
0927 05:06:34.225742 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.225748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.226548 (+ 800us) server_negotiation.cc:407] Connection header received
0927 05:06:34.226748 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.226752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.226832 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.226917 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:34.228445 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.229252 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.230650 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.231038 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.231180 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.234906 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:34.234911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:34.234914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:34.234945 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:34.237609 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.238304 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.238311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.238314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.238434 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.238839 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.238845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.238848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.239128 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:34.239307 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.239738 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.240002 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:06:34.240432 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.225494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:50336)
0927 05:06:34.226363 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.226427 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:34.226613 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:34.227018 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:34.227028 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:34.227868 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:34.228258 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.228273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.229399 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.229403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.230497 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.230512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.231218 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.231225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.231404 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.232499 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:34.232516 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:34.234714 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:34.237739 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.237748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.237756 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.238188 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.238544 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.238549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.238553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.238707 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.239275 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:34.239283 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:34.239457 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 05:06:34.239646 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 05:06:34.239996 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":440,"threads_started":1}
I20260927 05:06:34.241735 7265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:06:34 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485594, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260927 05:06:34.253731 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.242444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38783 (local address 127.0.0.1:44976)
0927 05:06:34.242615 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.242635 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:34.242724 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:34.242972 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:34.242975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:34.243259 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:34.243376 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.243397 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.244491 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.244511 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.245418 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.245428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.246081 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.246085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.246193 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.246686 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:34.246701 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:34.248574 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:34.251419 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.251423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.251425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.251761 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.252567 (+ 806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.252570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.252572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.252689 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.253113 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:34.253116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:34.253221 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 05:06:34.253324 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 05:06:34.253520 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 05:06:34.253790 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.242550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.0.97.61:38783)
0927 05:06:34.242707 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.242712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.242729 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:06:34.242803 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.242807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.242862 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.242938 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:34.243556 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.244355 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.245556 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.245954 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.246049 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.248739 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:34.248744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:34.248747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:34.248777 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:34.251300 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.251852 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.251856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.251858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.252472 (+ 614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.252777 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.252780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.252782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.253020 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:34.253124 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.253400 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.253571 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 05:06:34.254868 7265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:06:34 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485594, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 05:06:34.266386 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.255509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44619 (local address 127.0.0.1:44826)
0927 05:06:34.255714 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.255728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:34.255796 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:34.256109 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:34.256113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:34.256447 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:34.256601 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.256609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.257484 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.257488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.258260 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.258269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.258966 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.258971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.259116 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.259616 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:34.259643 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:34.261598 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:34.264419 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.264424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.264428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.264749 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.265037 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.265040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.265043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.265174 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.265642 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:34.265645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:34.265753 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 05:06:34.265856 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 05:06:34.266055 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 05:06:34.266461 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.255674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.0.97.60:44619)
0927 05:06:34.255809 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.255813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.255826 (+ 13us) server_negotiation.cc:407] Connection header received
0927 05:06:34.255862 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.255865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.255917 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.256049 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:34.256723 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.257353 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.258366 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.258752 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.258844 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.261747 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:34.261751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:34.261754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:34.261792 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:34.264308 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.264841 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.264844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.264847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.264942 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.265260 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.265262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.265264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.265531 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:34.265642 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.265930 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.266188 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":47}
I20260927 05:06:34.271135 7265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: Bootstrap starting.
I20260927 05:06:34.274718 7265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:34.275738 7265 log.cc:824] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:34.278352 7265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: No bootstrap required, opened a new log
I20260927 05:06:34.285152 7265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } }
I20260927 05:06:34.285490 7265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:34.285627 7265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182fdc6058254c2987252532c20577d8, State: Initialized, Role: FOLLOWER
I20260927 05:06:34.286160 7265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } }
I20260927 05:06:34.287093 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } } }
I20260927 05:06:34.287984 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:06:34.288517 7265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 05:06:34.298249 7282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:06:34.298388 7282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:06:34.301489 7112 master_runner.cc:428] Detected that this master 182fdc6058254c2987252532c20577d8 is joining an existing cluster
I20260927 05:06:34.301596 7112 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:44619
I20260927 05:06:34.324544 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.305853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.0.97.62:46281)
0927 05:06:34.306042 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.306047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.307013 (+ 966us) server_negotiation.cc:407] Connection header received
0927 05:06:34.307118 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.307124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.307206 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.307315 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:34.309170 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.310616 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.313446 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.314433 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.314557 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.318227 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:34.318233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:34.318238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:34.318280 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:34.321389 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.322177 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.322182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.322185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.322320 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.322743 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.322748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.322751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.323148 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:34.323287 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.323979 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.324200 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:06:34.325953 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.306032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:50344)
0927 05:06:34.306877 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.306907 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:34.307024 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:34.307335 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:34.307338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:34.308406 (+ 1068us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:34.308949 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.308967 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.310843 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.310852 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.313209 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.313234 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.314992 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.315001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.315254 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.316207 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:34.316239 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:34.317954 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:34.321542 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.321551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.321557 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.322042 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.322453 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.322458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.322461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.322635 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.323375 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:34.323382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:34.323631 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 05:06:34.325117 (+ 1486us) client_negotiation.cc:240] Negotiation successful
0927 05:06:34.325428 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":4,"thread_start_us":564,"threads_started":1}
I20260927 05:06:34.328783 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.305839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.0.97.61:38783)
0927 05:06:34.306021 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.306030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.312234 (+ 6204us) server_negotiation.cc:407] Connection header received
0927 05:06:34.312690 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.312696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.312774 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.312841 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:34.314392 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.315472 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.318544 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.319191 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.319286 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.321808 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:34.321813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:34.321818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:34.321855 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:34.325197 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.326006 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.326011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.326015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.326157 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.327129 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.327134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.327137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.327475 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:34.327603 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.328224 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.328444 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 05:06:34.330019 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.308303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38783 (local address 127.0.0.1:44978)
0927 05:06:34.312135 (+ 3832us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.312155 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:34.312519 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:34.313106 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:34.313113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:34.313843 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:34.314223 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.314241 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.315699 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.315704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.317405 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.317419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.319559 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.319565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.319750 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.320394 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:34.320416 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:34.321597 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:34.325348 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.325353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.325357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.325846 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.326692 (+ 846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.326699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.326703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.326998 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.327685 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:34.327690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:34.327900 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 05:06:34.329241 (+ 1341us) client_negotiation.cc:240] Negotiation successful
0927 05:06:34.329570 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3687,"spinlock_wait_cycles":91008,"thread_start_us":3940,"threads_started":1}
I20260927 05:06:34.337080 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.306087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.0.97.60:44619)
0927 05:06:34.306249 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.306253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.318465 (+ 12212us) server_negotiation.cc:407] Connection header received
0927 05:06:34.320212 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.320219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.320288 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.320355 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:34.321787 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.322829 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.325037 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.325739 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.325828 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.328605 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:34.328610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:34.328614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:34.328644 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:34.333591 (+ 4947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.334589 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.334597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.334603 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.334789 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.335248 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.335255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.335260 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.335668 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:34.335855 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.336425 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.336731 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":47}
I20260927 05:06:34.337103 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.308321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44619 (local address 127.0.0.1:44828)
0927 05:06:34.318374 (+ 10053us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.318392 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:34.320064 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:34.320553 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:34.320558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:34.321282 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:34.321643 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.321666 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.323047 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.323051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.324892 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.324907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.326148 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.326154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.326305 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.327303 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:34.327323 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:34.328404 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:34.333774 (+ 5370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.333781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.333788 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.334441 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.334960 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.334967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.334973 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.335131 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.335961 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:34.335967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:34.336162 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 05:06:34.336310 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 05:06:34.336616 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9922,"mutex_wait_us":264,"thread_start_us":9780,"threads_started":1}
Sep 27 05:06:34 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485591, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 05:06:34.344954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 6612
I20260927 05:06:34.352897 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.331676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44619 (local address 127.0.0.1:44830)
0927 05:06:34.331878 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.331896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:34.332039 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:34.336773 (+ 4734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:34.336780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:34.337233 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:34.337436 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.337448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.339867 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.339873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.341053 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.341065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.342031 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.342035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.342145 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.342701 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:34.342718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:34.345058 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:34.350146 (+ 5088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.350151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.350155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.350628 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.351076 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.351081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.351085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.351233 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.351872 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:34.351880 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:34.352065 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 05:06:34.352197 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 05:06:34.352477 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 05:06:34.354086 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.332100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.0.97.60:44619)
0927 05:06:34.336386 (+ 4286us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.336392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.336414 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:06:34.336507 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.336512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.336600 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.337579 (+ 979us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:34.338368 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.339719 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.341196 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.341875 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.342001 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.345280 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:34.345288 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:34.345292 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:34.345327 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:34.349956 (+ 4629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.350776 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.350782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.350786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.350951 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.351370 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.351376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.351380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.351750 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:34.353246 (+ 1496us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.353359 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.353585 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4157,"thread_start_us":4022,"threads_started":1}
I20260927 05:06:34.356451 6551 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:44619: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "182fdc6058254c2987252532c20577d8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 44619
}
attrs {
promote: true
}
}
dest_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99"
cas_config_opid_index: 5
I20260927 05:06:34.357354 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } }
I20260927 05:06:34.359025 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 182fdc6058254c2987252532c20577d8 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:34.359707 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 LEARNER]: Refusing update from remote peer 3169e443d11645c7b6a8f7c6f7fe2c99: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 05:06:34.360340 7293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 05:06:34.363986 7240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:34.369867 7108 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:44619
I20260927 05:06:34.371073 7240 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 FOLLOWER]: Refusing update from remote peer 3169e443d11645c7b6a8f7c6f7fe2c99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 05:06:34.373024 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:34.373163 7112 master.cc:537] Master@127.0.97.60:44619 shutting down...
I20260927 05:06:34.374670 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 182fdc6058254c2987252532c20577d8 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
Sep 27 05:06:34 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 05:06:34.377378 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:34.379566 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: true } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:34.379994 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: This master's current role is: LEARNER
I20260927 05:06:34.382272 7293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:34.385864 7293 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99: attempting to promote NON_VOTER cd7692436bd94dcc91d8823bb2692aba to VOTER
I20260927 05:06:34.387794 7293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } }
I20260927 05:06:34.392307 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.345870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43096 (local address 127.0.97.1:45703)
0927 05:06:34.346512 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.346520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.346549 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:06:34.346600 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.346606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.346908 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.347084 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:34.348966 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.349912 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.351613 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.352521 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.356160 (+ 3639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.382165 (+ 26005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:34.382188 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:34.382210 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:34.382268 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:34.388390 (+ 6122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.389360 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.389371 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.389376 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.389620 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.390141 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.390147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.390152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.390585 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:34.390851 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.391527 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.391837 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":333,"threads_started":1}
I20260927 05:06:34.395038 6735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:34.395040 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 LEARNER]: Refusing update from remote peer 3169e443d11645c7b6a8f7c6f7fe2c99: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 05:06:34.395678 7108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 05:06:34.398510 7293 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cd7692436bd94dcc91d8823bb2692aba (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:34.399202 7157 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cd7692436bd94dcc91d8823bb2692aba (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:34.402153 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:34.402465 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:34.402645 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:34.402877 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:34.403851 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:34.404162 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:06:34.411365 7112 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:34.411918 7112 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:34.412266 7112 tablet_replica.cc:330] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: stopping tablet replica
I20260927 05:06:34.420486 7112 master.cc:552] Master@127.0.97.60:44619 shutdown complete.
I20260927 05:06:34.420665 7112 master_runner.cc:305] Clearing existing system tablet
I20260927 05:06:34.421207 7112 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:34.421344 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:34.431202 7112 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.009s sys 0.000s
I20260927 05:06:34.433449 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.432854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44619 (local address 127.0.0.1:44834)
0927 05:06:34.433101 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.433239 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 05:06:34.434321 6537 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 -> Peer 182fdc6058254c2987252532c20577d8 (127.0.97.60:44619): Couldn't send request to peer 182fdc6058254c2987252532c20577d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44619: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 05:06:34.435268 7301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:34.435618 7112 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:34.435881 7112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:06:34.436028 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal
uuid: "182fdc6058254c2987252532c20577d8"
format_stamp: "Formatted at 2026-09-27 05:06:34 on dist-test-slave-hgs0"
server_key: "b09846fc61dc5bd163e93f647d482d18"
server_key_iv: "c1fc84e9a8200c47e249cf8d427cf176"
server_key_version: "encryptionkey@0"
I20260927 05:06:34.439307 7112 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:34.444409 7112 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 05:06:34.444598 7112 log.cc:1196] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 05:06:34.445034 7112 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: Deleting consensus metadata
I20260927 05:06:34.445291 7112 master_runner.cc:315] Copying system tablet from 127.0.97.62:46281
I20260927 05:06:34.479516 7112 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46281
I20260927 05:06:34.493825 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.480334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.0.97.62:46281)
0927 05:06:34.480545 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.480549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.481158 (+ 609us) server_negotiation.cc:407] Connection header received
0927 05:06:34.481370 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.481375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.481450 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.481554 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:34.482344 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.483076 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.484608 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.485108 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.485233 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.487615 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:34.487622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:34.487626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:34.487707 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:34.490686 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.491509 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.491514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.491520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.491655 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:34.492075 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:34.492080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:34.492084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:34.492592 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:34.492740 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.493217 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.493446 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:06:34.493917 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.480321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:50354)
0927 05:06:34.481051 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.481070 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:34.481245 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:34.481587 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:34.481592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:34.481989 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:34.482195 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.482204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.483221 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.483228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.484451 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.484464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.485263 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.485269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.485463 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.486666 (+ 1203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:34.486702 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:34.487431 (+ 729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:34.490808 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.490814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.490818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.491396 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.491763 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:34.491768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:34.491772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:34.491982 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:34.492805 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:34.492808 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:34.492963 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 05:06:34.493109 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 05:06:34.493408 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":471,"threads_started":1}
I20260927 05:06:34.494829 6571 tablet_copy_service.cc:140] P 3169e443d11645c7b6a8f7c6f7fe2c99: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 182fdc6058254c2987252532c20577d8 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:50354)
I20260927 05:06:34.495014 6571 tablet_copy_service.cc:161] P 3169e443d11645c7b6a8f7c6f7fe2c99: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 182fdc6058254c2987252532c20577d8 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:50354: session id = 182fdc6058254c2987252532c20577d8-00000000000000000000000000000000
I20260927 05:06:34.497520 6571 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 05:06:34.499087 7112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 05:06:34.504546 7112 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: tablet copy: Starting download of 0 data blocks...
I20260927 05:06:34.504838 7112 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: tablet copy: Starting download of 1 WAL segments...
I20260927 05:06:34.507371 7112 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 05:06:34.510566 6571 tablet_copy_service.cc:342] P 3169e443d11645c7b6a8f7c6f7fe2c99: Request end of tablet copy session 182fdc6058254c2987252532c20577d8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:50354
I20260927 05:06:34.510715 6571 tablet_copy_service.cc:433] P 3169e443d11645c7b6a8f7c6f7fe2c99: ending tablet copy session 182fdc6058254c2987252532c20577d8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 182fdc6058254c2987252532c20577d8
I20260927 05:06:34.520833 7112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:34.523415 7308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:34.524163 7309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:34.531785 7112 server_base.cc:1034] running on GCE node
W20260927 05:06:34.532577 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:34.533036 7112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:34.533679 7112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:34.535003 7112 hybrid_clock.cc:648] HybridClock initialized: now 1790485594534953 us; error 39 us; skew 500 ppm
I20260927 05:06:34.535362 7112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:34 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485594, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:34.537710 7112 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260927 05:06:34.537833 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 6612
I20260927 05:06:34.541280 7112 webserver.cc:460] Webserver started at http://127.0.97.60:40201/ using document root <none> and password file <none>
I20260927 05:06:34.541662 7112 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:34.541795 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:34.544927 7112 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 05:06:34.546895 7318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:34.547323 7112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 05:06:34.547487 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal
uuid: "182fdc6058254c2987252532c20577d8"
format_stamp: "Formatted at 2026-09-27 05:06:34 on dist-test-slave-hgs0"
server_key: "b09846fc61dc5bd163e93f647d482d18"
server_key_iv: "c1fc84e9a8200c47e249cf8d427cf176"
server_key_version: "encryptionkey@0"
I20260927 05:06:34.547729 7112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:34.557025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45703
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46281
--webserver_port=39013
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46281,127.0.97.61:38783
--tserver_master_addrs=127.0.97.62:46281,127.0.97.61:38783,127.0.97.60:44619 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:06:34.573144 7112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:34.577397 7112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:34.622701 7112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44619
I20260927 05:06:34.622740 7370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44619 every 8 connection(s)
I20260927 05:06:34.623684 7112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
I20260927 05:06:34.630438 7371 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:06:34.633013 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: Bootstrap starting.
I20260927 05:06:34.646054 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:06:34.646522 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: Bootstrap complete.
I20260927 05:06:34.647379 7371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } }
I20260927 05:06:34.647634 7371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 182fdc6058254c2987252532c20577d8, State: Initialized, Role: LEARNER
I20260927 05:06:34.647992 7371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } }
I20260927 05:06:34.648772 7374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:34.649119 7371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:34.649233 7374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: This master's current role is: LEARNER
I20260927 05:06:34.657430 7385 catalog_manager.cc:1259] Loaded cluster ID: c5b3015a0d3e49ebbdc03f7c087ab107
I20260927 05:06:34.657542 7385 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: loading cluster ID for follower catalog manager: success
I20260927 05:06:34.659859 7385 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: acquiring CA information for follower catalog manager: success
I20260927 05:06:34.661774 7385 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:06:34.791950 7386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.783723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.0.97.60:44619)
0927 05:06:34.784466 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:34.784472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:34.784492 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:06:34.784575 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:34.784583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:34.784676 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:34.784800 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:34.785349 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.787016 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.789426 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.790788 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.790957 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.791225 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:34.791370 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 05:06:34.791521 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":426,"threads_started":1}
I20260927 05:06:34.791954 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:34.783515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44619 (local address 127.0.0.1:44838)
0927 05:06:34.783775 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:34.783795 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:34.783923 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:34.784883 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:34.784891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:34.784941 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:34.785165 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.785181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.787195 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.787201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.789248 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:34.789264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:34.790993 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:34.791000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:34.791185 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:34.791220 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:06:34.791328 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 05:06:34.791527 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260927 05:06:34.793347 7346 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 05:06:34.794338 7374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3169e443d11645c7b6a8f7c6f7fe2c99. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:34.794692 7374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: This master's current role is: LEARNER
W20260927 05:06:34.940572 7323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:34.940855 7323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:34.941028 7323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:34.948438 7323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:34.948563 7323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:34.948738 7323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:34.959179 7323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39013
--tserver_master_addrs=127.0.97.62:46281,127.0.97.61:38783,127.0.97.60:44619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:34.960520 7323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:34.961797 7323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:34.975178 7391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:34.975224 7392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:34.975274 7394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:34.976176 7323 server_base.cc:1034] running on GCE node
I20260927 05:06:34.976846 7323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:34.977761 7323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:34.979064 7323 hybrid_clock.cc:648] HybridClock initialized: now 1790485594978998 us; error 51 us; skew 500 ppm
I20260927 05:06:34.979417 7323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:34 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485594, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:34.982664 7323 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 05:06:34.984172 7323 webserver.cc:460] Webserver started at http://127.0.97.1:39013/ using document root <none> and password file <none>
I20260927 05:06:34.984823 7323 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:34.984949 7323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:34.990454 7323 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:06:34.993376 7401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:34.995323 7323 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:06:34.995496 7323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "ec4f8eee7df847efb91934b3963549e8"
format_stamp: "Formatted at 2026-09-27 05:06:31 on dist-test-slave-hgs0"
server_key: "eefd2f07402f88464fca5e0978c2cd78"
server_key_iv: "0a7e9742ca08c2d37e89260e4b20235d"
server_key_version: "encryptionkey@0"
I20260927 05:06:34.996593 7323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:35.018095 7323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:35.023860 7323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:35.024226 7323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:35.025241 7323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:35.026717 7323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:35.026819 7323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:35.026907 7323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:35.026971 7323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:35.065834 7323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45703
I20260927 05:06:35.065934 7513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45703 every 8 connection(s)
I20260927 05:06:35.067420 7323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:35.071830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 7323
I20260927 05:06:35.072204 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 6746
Sep 27 05:06:35 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485594, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 05:06:35 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485594, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 05:06:35 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485594, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 05:06:35 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 05:06:35.127744 7386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.072277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40881 (local address 127.0.97.60:44619)
0927 05:06:35.072476 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:35.072482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:35.100660 (+ 28178us) server_negotiation.cc:407] Connection header received
0927 05:06:35.101723 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:35.101733 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:35.101854 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:35.101987 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:35.106519 (+ 4532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.107909 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.112707 (+ 4798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.113448 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.113577 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.119431 (+ 5854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:35.119438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:35.119444 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:35.119488 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:35.123060 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.124722 (+ 1662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.124728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.124731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.124881 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.125655 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.125662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.125668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.126017 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:35.126147 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:35.127175 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0927 05:06:35.127414 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:06:35.129336 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.072667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44619 (local address 127.0.97.1:40881)
0927 05:06:35.100526 (+ 27859us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:35.100551 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:35.100722 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:35.102594 (+ 1872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:35.102613 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:35.103840 (+ 1227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:35.106322 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.106336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.108209 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.108218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.112492 (+ 4274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.112507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.113654 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.113662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.113841 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.116085 (+ 2244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:35.116113 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:35.119173 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:35.123227 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.123232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.123238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.124577 (+ 1339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.125082 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.125087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.125090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.125513 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.126247 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:35.126256 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:35.126713 (+ 457us) client_negotiation.cc:769] Sending connection context
0927 05:06:35.127953 (+ 1240us) client_negotiation.cc:240] Negotiation successful
0927 05:06:35.128215 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27648,"mutex_wait_us":1775,"thread_start_us":27464,"threads_started":1}
I20260927 05:06:35.129554 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.070596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59933 (local address 127.0.97.62:46281)
0927 05:06:35.102442 (+ 31846us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:35.102448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:35.102465 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:06:35.102512 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:35.102516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:35.102587 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:35.102845 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:35.106218 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.107759 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.111416 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.112097 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.112206 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.119070 (+ 6864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:35.119077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:35.119081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:35.119123 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:35.122625 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.124194 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.124201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.124206 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.124364 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.125487 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.125493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.125496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.125900 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:35.126059 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:35.127601 (+ 1542us) server_negotiation.cc:299] Negotiation successful
0927 05:06:35.127831 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31736,"thread_start_us":31642,"threads_started":1}
I20260927 05:06:35.129308 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.073101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.1:59933)
0927 05:06:35.101283 (+ 28182us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:35.101303 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:35.101424 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:35.102760 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:35.102764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:35.105418 (+ 2654us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:35.105960 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.105993 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.108119 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.108128 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.111211 (+ 3083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.111249 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.112505 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.112512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.112872 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.114226 (+ 1354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:35.114254 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:35.118798 (+ 4544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:35.122777 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.122827 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.122848 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.124025 (+ 1177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.125048 (+ 1023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.125056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.125059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.125342 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.126926 (+ 1584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:35.126930 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:35.127066 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 05:06:35.127221 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 05:06:35.127732 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28044,"mutex_wait_us":8,"thread_start_us":22595,"threads_started":1}
I20260927 05:06:35.131405 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.073128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.0.97.2:33669)
0927 05:06:35.111582 (+ 38454us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:35.111594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:35.111625 (+ 31us) server_negotiation.cc:407] Connection header received
0927 05:06:35.111700 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:35.111706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:35.111905 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:35.112134 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:35.114061 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.115398 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.117156 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.117898 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.118046 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.123955 (+ 5909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:35.123973 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:35.123993 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:35.124036 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:35.127496 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.128688 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.128697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.128700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.128854 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.129278 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.129284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.129288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.129673 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:35.129961 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:35.130783 (+ 822us) server_negotiation.cc:299] Negotiation successful
0927 05:06:35.131039 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38240,"thread_start_us":38025,"threads_started":1}
I20260927 05:06:35.131457 7514 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44619
I20260927 05:06:35.131647 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.071324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38783 (local address 127.0.97.1:55615)
0927 05:06:35.096226 (+ 24902us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:35.096284 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:35.099442 (+ 3158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:35.112183 (+ 12741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:35.112187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:35.112738 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:35.113078 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.113092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.116149 (+ 3057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.116158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.117818 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.117837 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.118992 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.118998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.119180 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.119723 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:35.119747 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:35.122402 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:35.126890 (+ 4488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.126895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.126901 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.127361 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.128282 (+ 921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.128289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.128306 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.128526 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.129792 (+ 1266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:35.129796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:35.129935 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 05:06:35.130062 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 05:06:35.130355 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24523,"mutex_wait_us":31,"thread_start_us":29513,"threads_started":1}
I20260927 05:06:35.131778 7514 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:35.131892 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.070494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55615 (local address 127.0.97.61:38783)
0927 05:06:35.111767 (+ 41273us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:35.111774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:35.111799 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:06:35.111871 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:35.111876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:35.111998 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:35.113232 (+ 1234us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:35.113858 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.115970 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.118117 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.118774 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.119465 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.122621 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:35.122627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:35.122631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:35.122671 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:35.126731 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.128018 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.128023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.128027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.128153 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.128675 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.128679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.128682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.129066 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:35.131006 (+ 1940us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:35.131125 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 05:06:35.131312 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41138,"thread_start_us":41024,"threads_started":1}
I20260927 05:06:35.131958 7515 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46281
I20260927 05:06:35.132076 7515 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:35.132844 7516 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38783
I20260927 05:06:35.132911 7515 heartbeater.cc:511] Master 127.0.97.62:46281 requested a full tablet report, sending...
I20260927 05:06:35.132977 7516 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:35.133388 7516 heartbeater.cc:511] Master 127.0.97.61:38783 requested a full tablet report, sending...
I20260927 05:06:35.132885 7514 heartbeater.cc:511] Master 127.0.97.60:44619 requested a full tablet report, sending...
I20260927 05:06:35.134598 6869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:35.135546 7148 ts_manager.cc:194] Registered new tserver with Master: ec4f8eee7df847efb91934b3963549e8 (127.0.97.1:45703)
I20260927 05:06:35.135687 7336 ts_manager.cc:194] Registered new tserver with Master: ec4f8eee7df847efb91934b3963549e8 (127.0.97.1:45703)
I20260927 05:06:35.136554 6551 ts_manager.cc:194] Re-registered known tserver with Master: ec4f8eee7df847efb91934b3963549e8 (127.0.97.1:45703)
I20260927 05:06:35.137382 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59933
I20260927 05:06:35.224220 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 6746
I20260927 05:06:35.240562 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33669
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46281
--webserver_port=43631
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46281,127.0.97.61:38783
--tserver_master_addrs=127.0.97.62:46281,127.0.97.61:38783,127.0.97.60:44619 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:06:35.387581 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: true } } }
I20260927 05:06:35.388046 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:35.389358 7527 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99: attempting to promote NON_VOTER 182fdc6058254c2987252532c20577d8 to VOTER
I20260927 05:06:35.389986 7527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } }
I20260927 05:06:35.391984 7346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 LEARNER]: Refusing update from remote peer 3169e443d11645c7b6a8f7c6f7fe2c99: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 05:06:35.391978 7157 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Refusing update from remote peer 3169e443d11645c7b6a8f7c6f7fe2c99: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 05:06:35.392496 7527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 05:06:35.392904 7298 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 05:06:35.395726 7527 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 182fdc6058254c2987252532c20577d8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:35.396509 7157 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 182fdc6058254c2987252532c20577d8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:35.399720 7527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:35.400039 7527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:35.400089 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:35.400496 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:35.405681 7531 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:35.406172 7531 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:06:35.406625 7527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:35.406904 7527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:35.407233 7346 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 182fdc6058254c2987252532c20577d8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:35.412161 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:35.412525 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 05:06:35.568980 7526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:35.569259 7526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:35.569401 7526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:35.576769 7526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:35.576907 7526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:35.577101 7526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:35.587018 7526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43631
--tserver_master_addrs=127.0.97.62:46281,127.0.97.61:38783,127.0.97.60:44619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:35.588470 7526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:35.589792 7526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:35.604343 7538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:35.604390 7537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:35.604436 7540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:35.605692 7526 server_base.cc:1034] running on GCE node
I20260927 05:06:35.606338 7526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:35.607338 7526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:35.608572 7526 hybrid_clock.cc:648] HybridClock initialized: now 1790485595608517 us; error 48 us; skew 500 ppm
I20260927 05:06:35.608901 7526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:35 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:35.612156 7526 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 05:06:35.613723 7526 webserver.cc:460] Webserver started at http://127.0.97.2:43631/ using document root <none> and password file <none>
I20260927 05:06:35.614430 7526 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:35.614580 7526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:35.620338 7526 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:06:35.623368 7547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:35.624956 7526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:35.625135 7526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "1691dc7882c1481eb19f193d542bb678"
format_stamp: "Formatted at 2026-09-27 05:06:32 on dist-test-slave-hgs0"
server_key: "8d29f0c993d73e4295cf16d6c286796d"
server_key_iv: "e64fe76a16dbe8423028b9e289bb2d1c"
server_key_version: "encryptionkey@0"
I20260927 05:06:35.626142 7526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:35.672622 7526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:35.676865 7526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:35.677148 7526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:35.678256 7526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:35.679605 7526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:35.679724 7526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:35.679852 7526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:35.679965 7526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:35.713965 7526 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33669
I20260927 05:06:35.714051 7659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33669 every 8 connection(s)
I20260927 05:06:35.715411 7526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:35.721315 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 7526
I20260927 05:06:35.721693 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 6879
Sep 27 05:06:35 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 05:06:35 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 27 05:06:35 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 27 05:06:35 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485595, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260927 05:06:35.761919 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.723454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.0.97.3:40097)
0927 05:06:35.736469 (+ 13015us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:35.736483 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:35.737293 (+ 810us) server_negotiation.cc:407] Connection header received
0927 05:06:35.737415 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:35.737428 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:35.737873 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:35.738057 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:35.741448 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.743177 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.745062 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.745896 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.746029 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.753910 (+ 7881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:35.753928 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:35.753951 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:35.754009 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:35.757720 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.758656 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.758667 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.758673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.758839 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.759351 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.759358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.759364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.759792 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:35.761120 (+ 1328us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:35.761275 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 05:06:35.761543 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12818,"thread_start_us":12670,"threads_started":1}
I20260927 05:06:35.762197 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.727166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38783 (local address 127.0.97.2:36125)
0927 05:06:35.733538 (+ 6372us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:35.733561 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:35.738631 (+ 5070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:35.740633 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:35.740640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:35.741385 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:35.741887 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.741919 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.743407 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.743415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.745578 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.745598 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.746951 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.746960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.747377 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.748497 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:35.748529 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:35.752196 (+ 3667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:35.756589 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.756600 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.756619 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.757474 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.758013 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.758020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.758026 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.758269 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.759131 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:35.759142 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:35.759611 (+ 469us) client_negotiation.cc:769] Sending connection context
0927 05:06:35.759996 (+ 385us) client_negotiation.cc:240] Negotiation successful
0927 05:06:35.760576 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6234,"mutex_wait_us":94,"thread_start_us":4966,"threads_started":1}
I20260927 05:06:35.762960 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.725784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44619 (local address 127.0.97.2:45617)
0927 05:06:35.729020 (+ 3236us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:35.729085 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:35.738745 (+ 9660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:35.739861 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:35.739875 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:35.740769 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:35.742222 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.742234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.744773 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.744779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.750963 (+ 6184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.750976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.752059 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.752066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.752267 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.752875 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:35.752908 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:35.756056 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:35.759989 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.759993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.759996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.760464 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.760884 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.760890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.760894 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.761073 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.761858 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:35.761863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:35.762019 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 05:06:35.762196 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 05:06:35.762498 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2885,"mutex_wait_us":121,"thread_start_us":7262,"threads_started":1}
I20260927 05:06:35.763260 7002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:35.763593 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.718438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36125 (local address 127.0.97.61:38783)
0927 05:06:35.740154 (+ 21716us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:35.740161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:35.740183 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:06:35.740249 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:35.740257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:35.740351 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:35.741555 (+ 1204us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:35.742100 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.743212 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.745741 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.746418 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.746589 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.752616 (+ 6027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:35.752625 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:35.752633 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:35.752700 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:35.756405 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.757689 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.757695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.757698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.757868 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.758430 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.758436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.758440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.758880 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:35.759051 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:35.762917 (+ 3866us) server_negotiation.cc:299] Negotiation successful
0927 05:06:35.763173 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21576,"thread_start_us":22011,"threads_started":1}
I20260927 05:06:35.764397 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.718438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45617 (local address 127.0.97.60:44619)
0927 05:06:35.737171 (+ 18733us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:35.737182 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:35.737218 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:06:35.738952 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:35.738961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:35.739080 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:35.739214 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:35.743368 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.744588 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.751116 (+ 6528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.751599 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.751760 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.756259 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:35.756263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:35.756267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:35.756299 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:35.759795 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.760602 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.760608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.760612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.760746 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.761232 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.761238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.761242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.761600 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:35.761766 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:35.763703 (+ 1937us) server_negotiation.cc:299] Negotiation successful
0927 05:06:35.763930 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18557,"thread_start_us":18803,"threads_started":1}
I20260927 05:06:35.764882 7662 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38783
I20260927 05:06:35.765259 7662 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:35.765393 7660 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44619
I20260927 05:06:35.765573 7660 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:35.766163 7662 heartbeater.cc:511] Master 127.0.97.61:38783 requested a full tablet report, sending...
I20260927 05:06:35.766219 7660 heartbeater.cc:511] Master 127.0.97.60:44619 requested a full tablet report, sending...
I20260927 05:06:35.767903 7148 ts_manager.cc:194] Registered new tserver with Master: 1691dc7882c1481eb19f193d542bb678 (127.0.97.2:33669)
I20260927 05:06:35.768023 7336 ts_manager.cc:194] Registered new tserver with Master: 1691dc7882c1481eb19f193d542bb678 (127.0.97.2:33669)
I20260927 05:06:35.769838 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.726605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.2:38929)
0927 05:06:35.732458 (+ 5853us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:35.732486 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:35.738123 (+ 5637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:35.742444 (+ 4321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:35.742449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:35.742933 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:35.743140 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.743148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.744995 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.745002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.746407 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:35.746421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.748176 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.748188 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:35.748499 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.749624 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:35.749651 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:35.753451 (+ 3800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:35.760920 (+ 7469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.760928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.760931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.761427 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.767684 (+ 6257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:35.767691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:35.767694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:35.767870 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:35.768930 (+ 1060us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:35.768936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:35.769091 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 05:06:35.769207 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 05:06:35.769491 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5700,"mutex_wait_us":19,"thread_start_us":1918,"threads_started":1}
I20260927 05:06:35.769819 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:35.718400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38929 (local address 127.0.97.62:46281)
0927 05:06:35.742102 (+ 23702us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:35.742111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:35.742133 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:06:35.742200 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:35.742205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:35.742304 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:35.742418 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:35.743651 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.744873 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.747156 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:35.747967 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:35.748139 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:35.753670 (+ 5531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:35.753679 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:35.753687 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:35.753736 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:35.760485 (+ 6749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.763255 (+ 2770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.763262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.763266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.763437 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:35.768381 (+ 4944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:35.768388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:35.768391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:35.768800 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:35.768957 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:35.769386 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 05:06:35.769562 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23563,"thread_start_us":23413,"threads_started":1}
I20260927 05:06:35.770792 7661 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46281
I20260927 05:06:35.770944 7661 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:35.771400 7661 heartbeater.cc:511] Master 127.0.97.62:46281 requested a full tablet report, sending...
I20260927 05:06:35.772578 6551 ts_manager.cc:194] Re-registered known tserver with Master: 1691dc7882c1481eb19f193d542bb678 (127.0.97.2:33669)
I20260927 05:06:35.773823 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38929
I20260927 05:06:35.852977 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 6879
I20260927 05:06:35.869258 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40097
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46281
--webserver_port=42043
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46281,127.0.97.61:38783
--tserver_master_addrs=127.0.97.62:46281,127.0.97.61:38783,127.0.97.60:44619 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:06:36.139503 7515 heartbeater.cc:503] Master 127.0.97.62:46281 was elected leader, sending a full tablet report...
W20260927 05:06:36.204757 7673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:36.205075 7673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:36.205232 7673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:36.212532 7673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:36.212695 7673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:36.212872 7673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:36.222608 7673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=42043
--tserver_master_addrs=127.0.97.62:46281,127.0.97.61:38783,127.0.97.60:44619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:36.224079 7673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:36.225456 7673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:36.238775 7678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:36.238870 7679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:36.238875 7681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:36.239624 7673 server_base.cc:1034] running on GCE node
I20260927 05:06:36.240327 7673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:36.241323 7673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:36.242520 7673 hybrid_clock.cc:648] HybridClock initialized: now 1790485596242451 us; error 61 us; skew 500 ppm
I20260927 05:06:36.242844 7673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:36 dist-test-slave-hgs0 krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:36.245960 7673 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 05:06:36.247370 7673 webserver.cc:460] Webserver started at http://127.0.97.3:42043/ using document root <none> and password file <none>
I20260927 05:06:36.248127 7673 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:36.248278 7673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:36.253717 7673 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 05:06:36.256574 7688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:36.258011 7673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:36.258196 7673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "4d5a2a91d9a5430a9138d357ee7c1cff"
format_stamp: "Formatted at 2026-09-27 05:06:32 on dist-test-slave-hgs0"
server_key: "4046db49c57c6d64d0cd7634e313fc94"
server_key_iv: "9c8c640e8d89d058e62b48999cc7bd77"
server_key_version: "encryptionkey@0"
I20260927 05:06:36.259236 7673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:36.301067 7673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:36.305342 7673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:36.305637 7673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:36.306643 7673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:36.308061 7673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:36.308182 7673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:36.308310 7673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:36.308403 7673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:36.344386 7673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40097
I20260927 05:06:36.344446 7800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40097 every 8 connection(s)
I20260927 05:06:36.346340 7673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:36.350615 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 7673
Sep 27 05:06:36 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 27 05:06:36 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 27 05:06:36 dist-test-slave-hgs0 krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485596, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:36.391885 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.351200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58361 (local address 127.0.97.60:44619)
0927 05:06:36.352141 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.352147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.355525 (+ 3378us) server_negotiation.cc:407] Connection header received
0927 05:06:36.356902 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.356911 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.357047 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.357151 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:36.360406 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.362025 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.366023 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.366674 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.366809 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.379024 (+ 12215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:36.379031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:36.379037 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:36.379082 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:36.385340 (+ 6258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.386499 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.386507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.386511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.386683 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.387308 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.387316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.387323 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.387733 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:36.387879 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.391277 (+ 3398us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.391514 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":934,"threads_started":1}
I20260927 05:06:36.392561 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.350358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44619 (local address 127.0.97.3:58361)
0927 05:06:36.352643 (+ 2285us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:36.352709 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:36.356454 (+ 3745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:36.357880 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:36.357897 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:36.358953 (+ 1056us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:36.360104 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.360138 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.362297 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.362319 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.365839 (+ 3520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.365859 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.367119 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.367126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.367430 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.374826 (+ 7396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:36.374865 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:36.378716 (+ 3851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:36.385517 (+ 6801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:36.385544 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:36.385567 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:36.386335 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:36.386836 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:36.386841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:36.386845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:36.387155 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:36.387993 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:36.388007 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:36.388492 (+ 485us) client_negotiation.cc:769] Sending connection context
0927 05:06:36.388827 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 05:06:36.389861 (+ 1034us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1748,"thread_start_us":10677,"threads_started":1}
I20260927 05:06:36.394830 7801 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44619
I20260927 05:06:36.395174 7801 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:36.396546 7801 heartbeater.cc:511] Master 127.0.97.60:44619 requested a full tablet report, sending...
I20260927 05:06:36.398419 7336 ts_manager.cc:194] Registered new tserver with Master: 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097)
I20260927 05:06:36.403134 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.349613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46485 (local address 127.0.97.62:46281)
0927 05:06:36.351103 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.351110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.368647 (+ 17537us) server_negotiation.cc:407] Connection header received
0927 05:06:36.368845 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.368850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.368921 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.369037 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:36.369911 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.370859 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.372664 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.373335 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.373441 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.384617 (+ 11176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:36.384624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:36.384629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:36.384683 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:36.388970 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.392964 (+ 3994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.392970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.392973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.393137 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.394237 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.394241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.394245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.394606 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:36.394738 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.402513 (+ 7775us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.402762 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":284,"threads_started":1}
I20260927 05:06:36.403676 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.351602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38783 (local address 127.0.97.3:53217)
0927 05:06:36.360941 (+ 9339us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:36.360975 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:36.361134 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:36.362541 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:36.362546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:36.363093 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:36.363349 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.363357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.365404 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.365411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.366895 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.366920 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.368127 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.368135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.368367 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.376466 (+ 8099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:36.376490 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:36.394858 (+ 18368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:36.400688 (+ 5830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:36.400693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:36.400697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:36.401257 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:36.401782 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:36.401787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:36.401791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:36.401968 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:36.402653 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:36.402660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:36.402803 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 05:06:36.402969 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 05:06:36.403261 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9149,"thread_start_us":606,"threads_started":1}
I20260927 05:06:36.404510 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.352149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.3:46485)
0927 05:06:36.368523 (+ 16374us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:36.368552 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:36.368683 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:36.369055 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:36.369060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:36.369539 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:36.369756 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.369764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.371073 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.371078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.372516 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.372531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.373733 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.373741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.373959 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.380352 (+ 6393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:36.380382 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:36.384347 (+ 3965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:36.390949 (+ 6602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:36.390954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:36.390958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:36.391506 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:36.393289 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:36.393294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:36.393297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:36.393475 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:36.400748 (+ 7273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:36.400754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:36.400934 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 05:06:36.403378 (+ 2444us) client_negotiation.cc:240] Negotiation successful
0927 05:06:36.403704 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16218,"mutex_wait_us":47,"thread_start_us":15986,"threads_started":1}
I20260927 05:06:36.404577 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.349670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53217 (local address 127.0.97.61:38783)
0927 05:06:36.350306 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.350313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.361082 (+ 10769us) server_negotiation.cc:407] Connection header received
0927 05:06:36.361254 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.361260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.361342 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.361451 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:36.363672 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.365261 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.367079 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.367709 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.367843 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.395106 (+ 27263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:36.395117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:36.395122 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:36.395178 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:36.400539 (+ 5361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.401407 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.401413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.401416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.401646 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.402118 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.402123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.402126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.402463 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:36.402597 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.403984 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.404205 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":419,"threads_started":1}
I20260927 05:06:36.404829 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.370771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.0.97.62:46281)
0927 05:06:36.371297 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.371303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.372064 (+ 761us) server_negotiation.cc:407] Connection header received
0927 05:06:36.372289 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.372295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.372364 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.372452 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:36.374952 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.376101 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.378845 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.379509 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.379611 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.382975 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:36.382980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:36.382984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:36.383018 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:36.390793 (+ 7775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.393621 (+ 2828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.393629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.393633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.393764 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.401171 (+ 7407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.401179 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.401182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.401587 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:36.401824 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.404134 (+ 2310us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.404385 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":238,"threads_started":1}
I20260927 05:06:36.405383 7803 heartbeater.cc:347] Connected to a master server at 127.0.97.61:38783
I20260927 05:06:36.405396 7802 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46281
I20260927 05:06:36.405592 7803 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:36.405633 7802 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:36.406154 7802 heartbeater.cc:511] Master 127.0.97.62:46281 requested a full tablet report, sending...
I20260927 05:06:36.406816 7803 heartbeater.cc:511] Master 127.0.97.61:38783 requested a full tablet report, sending...
I20260927 05:06:36.407775 6550 ts_manager.cc:194] Re-registered known tserver with Master: 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097)
I20260927 05:06:36.407848 7148 ts_manager.cc:194] Registered new tserver with Master: 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097)
I20260927 05:06:36.409031 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46485
I20260927 05:06:36.411675 6550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 05:06:36.422506 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.371880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.0.97.60:44619)
0927 05:06:36.394466 (+ 22586us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.394472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.398837 (+ 4365us) server_negotiation.cc:407] Connection header received
0927 05:06:36.400835 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.400844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.400968 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.401056 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:36.408633 (+ 7577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.409716 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.411856 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.412430 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.412512 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.416236 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:36.416242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:36.416249 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:36.416302 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:36.419538 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.420361 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.420367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.420370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.420532 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.421002 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.421024 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.421027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.421377 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:36.421635 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.421942 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.422160 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22480,"thread_start_us":23386,"threads_started":1}
W20260927 05:06:36.422504 6550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:06:36.424664 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.377827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.0.97.61:38783)
0927 05:06:36.401678 (+ 23851us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.401687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.404432 (+ 2745us) server_negotiation.cc:407] Connection header received
0927 05:06:36.404487 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.404492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.405089 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.405213 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:36.409387 (+ 4174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.410884 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.412995 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.413619 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.413772 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.417573 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:36.417581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:36.417586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:36.417628 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:36.420783 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.422219 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.422226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.422229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.422375 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.422841 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.422847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.422851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.423181 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:36.423419 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.424149 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.424369 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22792,"thread_start_us":22578,"threads_started":1}
I20260927 05:06:36.469416 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.445203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45703 (local address 127.0.0.1:60954)
0927 05:06:36.448527 (+ 3324us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:36.448546 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:36.448652 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:36.461261 (+ 12609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:36.461267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:36.461298 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:36.461667 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.461680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.463727 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.463733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.466349 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.466364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.468441 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.468448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.468643 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.468680 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:06:36.468745 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 05:06:36.468898 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3161,"mutex_wait_us":299,"thread_start_us":431,"threads_started":1}
I20260927 05:06:36.470757 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.448895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.0.97.1:45703)
0927 05:06:36.460552 (+ 11657us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.460566 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.460610 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:06:36.460740 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.460749 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.461030 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.461278 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:36.461911 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.463477 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.466535 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.468189 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.469879 (+ 1690us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.470119 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.470205 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.470381 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11381,"thread_start_us":11183,"threads_started":1}
I20260927 05:06:36.473486 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.444721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40097 (local address 127.0.0.1:46234)
0927 05:06:36.445913 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:36.445935 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:36.446053 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:36.462935 (+ 16882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:36.462939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:36.462968 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:36.464496 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.464511 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.466771 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.466776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.469037 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.469054 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.472656 (+ 3602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.472665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.472869 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.472913 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 05:06:36.473013 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 05:06:36.473154 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":3492,"threads_started":1}
I20260927 05:06:36.474236 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.448553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.0.97.3:40097)
0927 05:06:36.461131 (+ 12578us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.461143 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.461188 (+ 45us) server_negotiation.cc:407] Connection header received
0927 05:06:36.461313 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.461324 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.461617 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.461852 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:36.464721 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.466561 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.470557 (+ 3996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.472129 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.472245 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.472451 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.473741 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.473928 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12329,"thread_start_us":12092,"threads_started":1}
I20260927 05:06:36.474809 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.448790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.0.97.2:33669)
0927 05:06:36.449455 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.449469 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.462742 (+ 13273us) server_negotiation.cc:407] Connection header received
0927 05:06:36.462936 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.462944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.463190 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.463396 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:36.467195 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.468775 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.471906 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.473321 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.473420 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.473795 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.474345 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.474544 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":314,"threads_started":1}
I20260927 05:06:36.475087 7449 tablet_service.cc:1467] Processing CreateTablet for tablet 764677f0b410441d93acf3df03874451 (DEFAULT_TABLE table=test-table [id=8198aea33c984ce2b1e18016bfafba39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:36.475859 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.448589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33669 (local address 127.0.0.1:37884)
0927 05:06:36.462589 (+ 14000us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:36.462608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:36.462765 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:36.463704 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:36.463711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:36.463741 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:36.466799 (+ 3058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.466810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.468971 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.468976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.471713 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.471732 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.473994 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.474003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.474151 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.474173 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 05:06:36.474285 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 05:06:36.475551 (+ 1266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13882,"mutex_wait_us":2609,"thread_start_us":13741,"threads_started":1}
I20260927 05:06:36.477106 7449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764677f0b410441d93acf3df03874451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:36.478690 7736 tablet_service.cc:1467] Processing CreateTablet for tablet 764677f0b410441d93acf3df03874451 (DEFAULT_TABLE table=test-table [id=8198aea33c984ce2b1e18016bfafba39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:36.479290 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 764677f0b410441d93acf3df03874451 (DEFAULT_TABLE table=test-table [id=8198aea33c984ce2b1e18016bfafba39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:36.480155 7736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764677f0b410441d93acf3df03874451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:36.480360 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764677f0b410441d93acf3df03874451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:36.491873 7838 tablet_bootstrap.cc:492] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: Bootstrap starting.
I20260927 05:06:36.493775 7839 tablet_bootstrap.cc:492] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: Bootstrap starting.
I20260927 05:06:36.496306 7838 tablet_bootstrap.cc:654] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:36.497819 7838 log.cc:824] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:36.498440 7839 tablet_bootstrap.cc:654] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:36.500177 7839 log.cc:824] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:36.502241 7840 tablet_bootstrap.cc:492] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: Bootstrap starting.
I20260927 05:06:36.509300 7839 tablet_bootstrap.cc:492] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: No bootstrap required, opened a new log
I20260927 05:06:36.509670 7839 ts_tablet_manager.cc:1397] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20260927 05:06:36.510715 7838 tablet_bootstrap.cc:492] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: No bootstrap required, opened a new log
I20260927 05:06:36.511053 7840 tablet_bootstrap.cc:654] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:36.511057 7838 ts_tablet_manager.cc:1397] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20260927 05:06:36.513024 7840 log.cc:824] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:36.516796 7840 tablet_bootstrap.cc:492] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: No bootstrap required, opened a new log
I20260927 05:06:36.517134 7840 ts_tablet_manager.cc:1397] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260927 05:06:36.520428 7839 raft_consensus.cc:348] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } } peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } }
I20260927 05:06:36.520581 7838 raft_consensus.cc:348] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } } peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } }
I20260927 05:06:36.520948 7839 raft_consensus.cc:374] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:36.520964 7838 raft_consensus.cc:374] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:36.521063 7839 raft_consensus.cc:729] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4f8eee7df847efb91934b3963549e8, State: Initialized, Role: FOLLOWER
I20260927 05:06:36.521060 7838 raft_consensus.cc:729] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1691dc7882c1481eb19f193d542bb678, State: Initialized, Role: FOLLOWER
I20260927 05:06:36.521729 7838 consensus_queue.cc:260] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } } peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } }
I20260927 05:06:36.521818 7839 consensus_queue.cc:260] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } } peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } }
I20260927 05:06:36.523679 7838 ts_tablet_manager.cc:1428] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260927 05:06:36.524145 7661 heartbeater.cc:503] Master 127.0.97.62:46281 was elected leader, sending a full tablet report...
I20260927 05:06:36.531523 7839 ts_tablet_manager.cc:1428] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: Time spent starting tablet: real 0.022s user 0.022s sys 0.000s
I20260927 05:06:36.533913 7840 raft_consensus.cc:348] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } } peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } }
I20260927 05:06:36.535523 7840 raft_consensus.cc:374] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:36.535640 7840 raft_consensus.cc:729] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5a2a91d9a5430a9138d357ee7c1cff, State: Initialized, Role: FOLLOWER
I20260927 05:06:36.536374 7840 consensus_queue.cc:260] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } } peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } }
I20260927 05:06:36.537429 7802 heartbeater.cc:503] Master 127.0.97.62:46281 was elected leader, sending a full tablet report...
I20260927 05:06:36.538538 7840 ts_tablet_manager.cc:1428] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: Time spent starting tablet: real 0.021s user 0.012s sys 0.004s
W20260927 05:06:36.599928 7517 tablet.cc:2369] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:06:36.602136 7804 tablet.cc:2369] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:06:36.735109 7663 tablet.cc:2369] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:06:36.745050 7845 raft_consensus.cc:482] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:36.745365 7845 raft_consensus.cc:504] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } } peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } }
I20260927 05:06:36.747081 7845 leader_election.cc:288] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097), 1691dc7882c1481eb19f193d542bb678 (127.0.97.2:33669)
I20260927 05:06:36.754920 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.747479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38977 (local address 127.0.97.3:40097)
0927 05:06:36.747669 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.747692 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.748343 (+ 651us) server_negotiation.cc:407] Connection header received
0927 05:06:36.748616 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.748622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.748718 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.748788 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:36.749877 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.750722 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.753166 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.753955 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.754117 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.754226 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.754508 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.754635 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:06:36.755107 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.747491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40097 (local address 127.0.97.1:38977)
0927 05:06:36.748232 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:36.748254 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:36.748425 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:36.748999 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:36.749002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:36.749022 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:36.749700 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.749709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.750845 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.750853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.752996 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.753010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.754174 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.754179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.754344 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.754370 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 05:06:36.754480 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 05:06:36.754623 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":1,"thread_start_us":303,"threads_started":1}
I20260927 05:06:36.755406 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.748511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50081 (local address 127.0.97.2:33669)
0927 05:06:36.748728 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.748734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.749515 (+ 781us) server_negotiation.cc:407] Connection header received
0927 05:06:36.749737 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.749742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.749813 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.750130 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:36.750709 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.751469 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.753560 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.754423 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.754577 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.754691 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.755023 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.755152 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:06:36.755487 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.748326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33669 (local address 127.0.97.1:50081)
0927 05:06:36.748958 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:36.748982 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:36.749592 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:36.750136 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:36.750140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:36.750164 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:36.750507 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.750522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.751607 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.751614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.753391 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:36.753404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.754660 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.754666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:36.754857 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.754881 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 05:06:36.754996 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 05:06:36.755134 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":319,"threads_started":1}
I20260927 05:06:36.756340 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764677f0b410441d93acf3df03874451" candidate_uuid: "ec4f8eee7df847efb91934b3963549e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" is_pre_election: true
I20260927 05:06:36.756934 7756 raft_consensus.cc:2454] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4f8eee7df847efb91934b3963549e8 in term 0.
I20260927 05:06:36.756784 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764677f0b410441d93acf3df03874451" candidate_uuid: "ec4f8eee7df847efb91934b3963549e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1691dc7882c1481eb19f193d542bb678" is_pre_election: true
I20260927 05:06:36.757899 7402 leader_election.cc:302] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d5a2a91d9a5430a9138d357ee7c1cff, ec4f8eee7df847efb91934b3963549e8; no voters:
I20260927 05:06:36.758010 7615 raft_consensus.cc:2454] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4f8eee7df847efb91934b3963549e8 in term 0.
I20260927 05:06:36.758414 7845 raft_consensus.cc:2781] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:06:36.758638 7845 raft_consensus.cc:482] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:06:36.758792 7845 raft_consensus.cc:3037] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:36.761682 7845 raft_consensus.cc:504] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } } peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } }
I20260927 05:06:36.762543 7845 leader_election.cc:288] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [CANDIDATE]: Term 1 election: Requested vote from peers 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097), 1691dc7882c1481eb19f193d542bb678 (127.0.97.2:33669)
I20260927 05:06:36.763224 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764677f0b410441d93acf3df03874451" candidate_uuid: "ec4f8eee7df847efb91934b3963549e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff"
I20260927 05:06:36.763520 7756 raft_consensus.cc:3037] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:36.763554 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764677f0b410441d93acf3df03874451" candidate_uuid: "ec4f8eee7df847efb91934b3963549e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1691dc7882c1481eb19f193d542bb678"
I20260927 05:06:36.763847 7615 raft_consensus.cc:3037] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:36.766289 7756 raft_consensus.cc:2454] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4f8eee7df847efb91934b3963549e8 in term 1.
I20260927 05:06:36.766489 7615 raft_consensus.cc:2454] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4f8eee7df847efb91934b3963549e8 in term 1.
I20260927 05:06:36.766829 7402 leader_election.cc:302] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d5a2a91d9a5430a9138d357ee7c1cff, ec4f8eee7df847efb91934b3963549e8; no voters:
I20260927 05:06:36.767227 7845 raft_consensus.cc:2781] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:36.767751 7845 raft_consensus.cc:686] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 1 LEADER]: Becoming Leader. State: Replica: ec4f8eee7df847efb91934b3963549e8, State: Running, Role: LEADER
I20260927 05:06:36.768409 7845 consensus_queue.cc:237] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } } peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } }
I20260927 05:06:36.774367 6551 catalog_manager.cc:5495] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec4f8eee7df847efb91934b3963549e8 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ec4f8eee7df847efb91934b3963549e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } health_report { overall_health: UNKNOWN } } }
I20260927 05:06:36.831223 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.824708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.0.97.1:45703)
0927 05:06:36.825093 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.825109 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.825136 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:06:36.825254 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.825259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.825376 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.825594 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:06:36.826025 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.826833 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.828474 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.829030 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.829157 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.829899 (+ 742us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:06:36.830516 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:06:36.830653 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.830830 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.830921 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:06:36.840737 7756 raft_consensus.cc:1261] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff [term 1 FOLLOWER]: Refusing update from remote peer ec4f8eee7df847efb91934b3963549e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:36.840756 7615 raft_consensus.cc:1261] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678 [term 1 FOLLOWER]: Refusing update from remote peer ec4f8eee7df847efb91934b3963549e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:36.841807 7845 consensus_queue.cc:1035] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:36.842149 7849 consensus_queue.cc:1035] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:36.856763 7852 mvcc.cc:205] Tried to move back new op lower bound from 7333829004647776256 to 7333829004367511552. Current Snapshot: MvccSnapshot[applied={T|T < 7333829004647776256}]
I20260927 05:06:36.879897 6551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:36.880234 6551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:36.886791 6551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3169e443d11645c7b6a8f7c6f7fe2c99: Sending DeleteTablet for 3 replicas of tablet 764677f0b410441d93acf3df03874451
I20260927 05:06:36.888178 7449 tablet_service.cc:1514] Processing DeleteTablet for tablet 764677f0b410441d93acf3df03874451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:36 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60954
I20260927 05:06:36.888177 7736 tablet_service.cc:1514] Processing DeleteTablet for tablet 764677f0b410441d93acf3df03874451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:36 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46234
I20260927 05:06:36.889146 7861 tablet_replica.cc:330] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: stopping tablet replica
I20260927 05:06:36.889644 7861 raft_consensus.cc:2229] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:36.889633 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet 764677f0b410441d93acf3df03874451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:36 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37884
I20260927 05:06:36.890173 7861 raft_consensus.cc:2258] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:36.891004 7862 tablet_replica.cc:330] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: stopping tablet replica
I20260927 05:06:36.891461 7862 raft_consensus.cc:2229] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:06:36.892685 7862 raft_consensus.cc:2258] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:36.897471 7862 ts_tablet_manager.cc:1905] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:36.895341 7861 ts_tablet_manager.cc:1905] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:36.905148 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 6519
I20260927 05:06:36.906566 6581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:36.910372 7862 ts_tablet_manager.cc:1918] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:36.910614 7862 log.cc:1196] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/764677f0b410441d93acf3df03874451
I20260927 05:06:36.911026 7863 tablet_replica.cc:330] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: stopping tablet replica
I20260927 05:06:36.911279 7862 ts_tablet_manager.cc:1939] T 764677f0b410441d93acf3df03874451 P ec4f8eee7df847efb91934b3963549e8: Deleting consensus metadata
I20260927 05:06:36.911667 7863 raft_consensus.cc:2229] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:36.912923 7863 raft_consensus.cc:2258] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:36.915925 7861 ts_tablet_manager.cc:1918] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:36.916203 7861 log.cc:1196] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/764677f0b410441d93acf3df03874451
I20260927 05:06:36.916764 7861 ts_tablet_manager.cc:1939] T 764677f0b410441d93acf3df03874451 P 4d5a2a91d9a5430a9138d357ee7c1cff: Deleting consensus metadata
I20260927 05:06:36.920197 7863 ts_tablet_manager.cc:1905] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:36.929183 7863 ts_tablet_manager.cc:1918] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:36.929478 7863 log.cc:1196] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/764677f0b410441d93acf3df03874451
I20260927 05:06:36.930138 7863 ts_tablet_manager.cc:1939] T 764677f0b410441d93acf3df03874451 P 1691dc7882c1481eb19f193d542bb678: Deleting consensus metadata
I20260927 05:06:36.933868 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.892998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.0.97.61:38783)
0927 05:06:36.893465 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.893475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.903535 (+ 10060us) server_negotiation.cc:407] Connection header received
0927 05:06:36.906558 (+ 3023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.906566 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.906670 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.906766 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:36.913739 (+ 6973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.915253 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.921475 (+ 6222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.922138 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.922249 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.926048 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:36.926056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:36.926061 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:36.926109 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:36.930011 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.931083 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.931091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.931096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.931276 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.931815 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.931821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.931825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.932275 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:36.932552 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.933131 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.933451 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260927 05:06:36.953073 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:36.903725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.0.97.60:44619)
0927 05:06:36.903916 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:36.903922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:36.933928 (+ 30006us) server_negotiation.cc:407] Connection header received
0927 05:06:36.934212 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:36.934218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:36.934309 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:36.934380 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:36.936765 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.937994 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.940015 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:36.940708 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:36.940854 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:36.944578 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:36.944586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:36.944591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:36.944643 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:36.948866 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.949928 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.949935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.949939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.950115 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:36.950642 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:36.950648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:36.950652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:36.951100 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:36.951430 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:36.952161 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 05:06:36.952673 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:06:37.057613 6536 catalog_manager.cc:4841] TS ec4f8eee7df847efb91934b3963549e8 (127.0.97.1:45703): tablet 764677f0b410441d93acf3df03874451 (table test-table [id=8198aea33c984ce2b1e18016bfafba39]) successfully deleted
I20260927 05:06:37.058177 6535 catalog_manager.cc:4841] TS 1691dc7882c1481eb19f193d542bb678 (127.0.97.2:33669): tablet 764677f0b410441d93acf3df03874451 (table test-table [id=8198aea33c984ce2b1e18016bfafba39]) successfully deleted
I20260927 05:06:37.058490 6535 catalog_manager.cc:4841] TS 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097): tablet 764677f0b410441d93acf3df03874451 (table test-table [id=8198aea33c984ce2b1e18016bfafba39]) successfully deleted
I20260927 05:06:37.058570 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 6519
I20260927 05:06:37.120047 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:37.094271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.0.97.60:44619)
0927 05:06:37.094526 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:37.094535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:37.101094 (+ 6559us) server_negotiation.cc:407] Connection header received
0927 05:06:37.101458 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:37.101465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:37.101561 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:37.101638 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:37.104506 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:37.105785 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:37.107371 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:37.108020 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:37.108156 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:37.111800 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:37.111808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:37.111814 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:37.111863 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:37.116190 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:37.117250 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:37.117258 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:37.117262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:37.117450 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:37.118031 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:37.118037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:37.118042 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:37.118475 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:37.118712 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:37.119344 (+ 632us) server_negotiation.cc:299] Negotiation successful
0927 05:06:37.119626 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:06:37.120915 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:37.101015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.0.97.61:38783)
0927 05:06:37.101213 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:37.101222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:37.103512 (+ 2290us) server_negotiation.cc:407] Connection header received
0927 05:06:37.103838 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:37.103845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:37.103987 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:37.104106 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:37.106332 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:37.107574 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:37.109571 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:37.110317 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:37.110471 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:37.113804 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:37.113810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:37.113814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:37.113855 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:37.117424 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:37.118220 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:37.118227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:37.118231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:37.118451 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:37.118956 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:37.118962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:37.118966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:37.119409 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:37.119663 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:37.120286 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 05:06:37.120524 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:06:37.870618 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:37.869495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.3:50233)
0927 05:06:37.870231 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:37.870404 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":484,"threads_started":1}
W20260927 05:06:37.871088 7802 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46281 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
I20260927 05:06:37.871137 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:37.869779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.1:37089)
0927 05:06:37.870607 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:37.870976 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":605,"threads_started":1}
W20260927 05:06:37.871625 7515 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46281 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
I20260927 05:06:37.874464 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:37.872339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.2:49267)
0927 05:06:37.874032 (+ 1693us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:37.874186 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1570,"thread_start_us":1421,"threads_started":1}
W20260927 05:06:37.874992 7661 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46281 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
I20260927 05:06:38.387161 7875 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3169e443d11645c7b6a8f7c6f7fe2c99)
I20260927 05:06:38.387547 7875 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } }
I20260927 05:06:38.388022 7876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3169e443d11645c7b6a8f7c6f7fe2c99)
I20260927 05:06:38.388343 7876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } }
I20260927 05:06:38.389863 7876 leader_election.cc:288] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281), cd7692436bd94dcc91d8823bb2692aba (127.0.97.61:38783)
I20260927 05:06:38.390622 7875 leader_election.cc:288] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281), 182fdc6058254c2987252532c20577d8 (127.0.97.60:44619)
I20260927 05:06:38.392129 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.389845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51214)
0927 05:06:38.390468 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.390679 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"threads_started":1}
I20260927 05:06:38.392910 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.389915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51222)
0927 05:06:38.391451 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.391681 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1374,"thread_start_us":1182,"threads_started":1}
W20260927 05:06:38.393323 7133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
W20260927 05:06:38.393324 7320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
I20260927 05:06:38.401691 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.400790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51226)
0927 05:06:38.401045 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.401320 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 05:06:38.402101 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.392525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.0.97.61:38783)
0927 05:06:38.393387 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:38.393394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:38.393422 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:06:38.393502 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:38.393508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:38.393599 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:38.394434 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:38.394745 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:38.396163 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:38.398902 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:38.400460 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:38.401392 (+ 932us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:38.401546 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:38.401603 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 05:06:38.401754 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":495,"threads_started":1}
I20260927 05:06:38.402097 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.390281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38783 (local address 127.0.0.1:57814)
0927 05:06:38.392069 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.392478 (+ 409us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:38.392787 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:38.393779 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:38.393787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:38.393817 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:38.394232 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:38.394248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:38.396308 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:38.396314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:38.398700 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:38.398714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:38.400682 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:38.400688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:38.400934 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:38.400971 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:06:38.401052 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 05:06:38.401207 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"thread_start_us":1417,"threads_started":1}
W20260927 05:06:38.402235 7133 leader_election.cc:334] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
I20260927 05:06:38.403133 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.392057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.0.97.60:44619)
0927 05:06:38.393455 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:38.393461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:38.393476 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:06:38.393532 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:38.393537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:38.393601 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:38.393722 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:38.394539 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:38.396128 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:38.398884 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:38.400460 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:38.400613 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:38.402510 (+ 1897us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:38.402571 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 05:06:38.402718 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":1774,"threads_started":1}
I20260927 05:06:38.401719 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.389915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44619 (local address 127.0.0.1:37274)
0927 05:06:38.392505 (+ 2590us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.392525 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:38.392694 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:38.393998 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:38.394006 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:38.394043 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:38.394286 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:38.394302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:38.396317 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:38.396322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:38.398667 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:38.398688 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:38.400843 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:38.400849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:38.401070 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:38.401105 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:06:38.401185 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 05:06:38.401342 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2469,"thread_start_us":385,"threads_started":1}
I20260927 05:06:38.404186 7157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "182fdc6058254c2987252532c20577d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cd7692436bd94dcc91d8823bb2692aba" is_pre_election: true
I20260927 05:06:38.403928 7346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd7692436bd94dcc91d8823bb2692aba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "182fdc6058254c2987252532c20577d8" is_pre_election: true
I20260927 05:06:38.404392 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.403718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51228)
0927 05:06:38.404113 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.404206 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260927 05:06:38.404565 7157 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182fdc6058254c2987252532c20577d8 in term 1.
I20260927 05:06:38.404567 7346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7692436bd94dcc91d8823bb2692aba in term 1.
W20260927 05:06:38.404620 7320 leader_election.cc:334] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
I20260927 05:06:38.405090 7322 leader_election.cc:302] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 182fdc6058254c2987252532c20577d8, cd7692436bd94dcc91d8823bb2692aba; no voters: 3169e443d11645c7b6a8f7c6f7fe2c99
I20260927 05:06:38.405104 7134 leader_election.cc:302] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 182fdc6058254c2987252532c20577d8, cd7692436bd94dcc91d8823bb2692aba; no voters: 3169e443d11645c7b6a8f7c6f7fe2c99
I20260927 05:06:38.405500 7876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 05:06:38.405643 7876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3169e443d11645c7b6a8f7c6f7fe2c99)
I20260927 05:06:38.405735 7876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:06:38.405812 7875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 05:06:38.405920 7875 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3169e443d11645c7b6a8f7c6f7fe2c99)
I20260927 05:06:38.406010 7875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:06:38.409066 7875 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } }
I20260927 05:06:38.409052 7876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } }
I20260927 05:06:38.409861 7875 leader_election.cc:288] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 2 election: Requested vote from peers 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281), 182fdc6058254c2987252532c20577d8 (127.0.97.60:44619)
I20260927 05:06:38.410439 7346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd7692436bd94dcc91d8823bb2692aba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "182fdc6058254c2987252532c20577d8"
I20260927 05:06:38.410693 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.410153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51238)
0927 05:06:38.410314 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.410436 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 05:06:38.410835 7346 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cd7692436bd94dcc91d8823bb2692aba in current term 2: Already voted for candidate 182fdc6058254c2987252532c20577d8 in this term.
I20260927 05:06:38.411226 7157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "182fdc6058254c2987252532c20577d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cd7692436bd94dcc91d8823bb2692aba"
I20260927 05:06:38.411607 7157 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 182fdc6058254c2987252532c20577d8 in current term 2: Already voted for candidate cd7692436bd94dcc91d8823bb2692aba in this term.
I20260927 05:06:38.412021 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.410153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51240)
0927 05:06:38.411670 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.411794 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1372}
I20260927 05:06:38.412045 7876 leader_election.cc:288] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [CANDIDATE]: Term 2 election: Requested vote from peers 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281), cd7692436bd94dcc91d8823bb2692aba (127.0.97.61:38783)
I20260927 05:06:38.413226 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.412828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51280)
0927 05:06:38.412976 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.413064 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 05:06:38.413625 7320 leader_election.cc:334] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
I20260927 05:06:38.413810 7320 leader_election.cc:302] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 182fdc6058254c2987252532c20577d8; no voters: 3169e443d11645c7b6a8f7c6f7fe2c99, cd7692436bd94dcc91d8823bb2692aba
I20260927 05:06:38.414143 7876 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 05:06:38.414435 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.414017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51292)
0927 05:06:38.414181 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.414280 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 05:06:38.414656 7133 leader_election.cc:334] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
I20260927 05:06:38.414808 7133 leader_election.cc:302] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd7692436bd94dcc91d8823bb2692aba; no voters: 182fdc6058254c2987252532c20577d8, 3169e443d11645c7b6a8f7c6f7fe2c99
I20260927 05:06:38.415135 7875 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 05:06:38.872733 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.871966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.3:54971)
0927 05:06:38.872471 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.872563 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":259,"threads_started":1}
I20260927 05:06:38.873119 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.872351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.1:47281)
0927 05:06:38.872831 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.872969 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
I20260927 05:06:38.876567 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:38.875620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.2:56099)
0927 05:06:38.876300 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:38.876392 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":482,"threads_started":1}
I20260927 05:06:39.874780 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:39.873766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.3:51813)
0927 05:06:39.874394 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:39.874525 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":317,"threads_started":1}
I20260927 05:06:39.875169 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:39.874093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.1:33097)
0927 05:06:39.874702 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:39.874783 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":373,"threads_started":1}
I20260927 05:06:39.878187 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:39.877437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.2:48719)
0927 05:06:39.877956 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:39.878040 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":290,"threads_started":1}
I20260927 05:06:40.367877 7891 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:40.368201 7891 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } }
I20260927 05:06:40.368781 7891 leader_election.cc:288] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281), 182fdc6058254c2987252532c20577d8 (127.0.97.60:44619)
I20260927 05:06:40.369434 7346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd7692436bd94dcc91d8823bb2692aba" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "182fdc6058254c2987252532c20577d8" is_pre_election: true
I20260927 05:06:40.369686 7346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7692436bd94dcc91d8823bb2692aba in term 2.
I20260927 05:06:40.370033 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.369063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51532)
0927 05:06:40.369714 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.369828 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":356,"threads_started":1}
I20260927 05:06:40.370213 7134 leader_election.cc:302] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182fdc6058254c2987252532c20577d8, cd7692436bd94dcc91d8823bb2692aba; no voters:
I20260927 05:06:40.370512 7891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 05:06:40.370713 7891 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:06:40.370837 7891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 2 FOLLOWER]: Advancing to term 3
I20260927 05:06:40.373155 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.372800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51538)
0927 05:06:40.372954 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.373028 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:06:40.373342 7133 leader_election.cc:334] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
I20260927 05:06:40.374042 7891 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } }
I20260927 05:06:40.374598 7891 leader_election.cc:288] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 3 election: Requested vote from peers 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281), 182fdc6058254c2987252532c20577d8 (127.0.97.60:44619)
I20260927 05:06:40.375201 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.374746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51554)
0927 05:06:40.374934 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.375032 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 05:06:40.375196 7346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd7692436bd94dcc91d8823bb2692aba" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "182fdc6058254c2987252532c20577d8"
I20260927 05:06:40.375464 7346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 2 FOLLOWER]: Advancing to term 3
I20260927 05:06:40.376292 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.375983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51566)
0927 05:06:40.376123 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.376186 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:06:40.376475 7133 leader_election.cc:334] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
I20260927 05:06:40.377916 7346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7692436bd94dcc91d8823bb2692aba in term 3.
I20260927 05:06:40.378448 7134 leader_election.cc:302] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 182fdc6058254c2987252532c20577d8, cd7692436bd94dcc91d8823bb2692aba; no voters: 3169e443d11645c7b6a8f7c6f7fe2c99
I20260927 05:06:40.378777 7891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 3 FOLLOWER]: Leader election won for term 3
I20260927 05:06:40.379261 7891 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [term 3 LEADER]: Becoming Leader. State: Replica: cd7692436bd94dcc91d8823bb2692aba, State: Running, Role: LEADER
I20260927 05:06:40.379703 7891 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } }
I20260927 05:06:40.382160 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7692436bd94dcc91d8823bb2692aba. Latest consensus state: current_term: 3 leader_uuid: "cd7692436bd94dcc91d8823bb2692aba" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:40.382571 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:40.383457 7896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:40.388592 7896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:40.389667 7896 catalog_manager.cc:1259] Loaded cluster ID: c5b3015a0d3e49ebbdc03f7c087ab107
I20260927 05:06:40.389765 7896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:40.391204 7896 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:40.392294 7896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Loaded TSK: 0
I20260927 05:06:40.393252 7896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:06:40.414207 7148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:06:40.422043 7148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:06:40.426672 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.426192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51572)
0927 05:06:40.426366 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.426470 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 05:06:40.426744 7346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [term 3 FOLLOWER]: Refusing update from remote peer cd7692436bd94dcc91d8823bb2692aba: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 05:06:40.427521 7894 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 05:06:40.427850 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.427440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51588)
0927 05:06:40.427605 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.427682 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:06:40.428076 7133 consensus_peers.cc:487] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba -> Peer 3169e443d11645c7b6a8f7c6f7fe2c99 (127.0.97.62:46281): Couldn't send request to peer 3169e443d11645c7b6a8f7c6f7fe2c99. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 05:06:40.431365 7898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7692436bd94dcc91d8823bb2692aba. Latest consensus state: current_term: 3 leader_uuid: "cd7692436bd94dcc91d8823bb2692aba" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:40.431687 7898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:06:40.433229 7894 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "cd7692436bd94dcc91d8823bb2692aba" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:40.433584 7894 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:40.433822 7898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "cd7692436bd94dcc91d8823bb2692aba" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3169e443d11645c7b6a8f7c6f7fe2c99" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46281 } } peers { permanent_uuid: "cd7692436bd94dcc91d8823bb2692aba" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38783 } attrs { promote: false } } peers { permanent_uuid: "182fdc6058254c2987252532c20577d8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44619 } attrs { promote: false } } }
I20260927 05:06:40.434167 7898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 182fdc6058254c2987252532c20577d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:06:40.453861 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.443595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45703 (local address 127.0.0.1:60970)
0927 05:06:40.443760 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.443782 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:40.443917 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:40.444560 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:40.444566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:40.444594 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:40.444859 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:40.444872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.446985 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.446994 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:40.449833 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:40.449857 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.452616 (+ 2759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.452628 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:40.452925 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:40.452981 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 05:06:40.453078 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 05:06:40.453308 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 05:06:40.455293 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.443712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.0.97.1:45703)
0927 05:06:40.444240 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:40.444246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:40.444266 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:06:40.444338 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:40.444343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:40.444412 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:40.445210 (+ 798us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:40.445465 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.446827 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.450114 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.452278 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.454367 (+ 2089us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:40.454560 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:40.454634 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 05:06:40.454853 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":756,"threads_started":1}
I20260927 05:06:40.456404 7449 tablet_service.cc:1467] Processing CreateTablet for tablet c6d562fde4374b76bb65cb81262b0c1e (DEFAULT_TABLE table=test-table [id=92276af5269b447280f89d8324c07266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:40.457039 7449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d562fde4374b76bb65cb81262b0c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:40.459563 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.445314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.0.97.3:40097)
0927 05:06:40.445809 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:40.445815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:40.452727 (+ 6912us) server_negotiation.cc:407] Connection header received
0927 05:06:40.452989 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:40.452994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:40.453075 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:40.453194 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:40.453804 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.454759 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.457460 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.458559 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.458704 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:40.458811 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:40.459094 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 05:06:40.459230 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":560,"threads_started":1}
I20260927 05:06:40.459563 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.444931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40097 (local address 127.0.0.1:46250)
0927 05:06:40.452608 (+ 7677us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.452626 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:40.452788 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:40.453209 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:40.453215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:40.453242 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:40.453628 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:40.453642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.455164 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.455170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:40.457276 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:40.457290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.458728 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.458734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:40.458931 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:40.458957 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 05:06:40.459066 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 05:06:40.459206 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7472,"thread_start_us":7271,"threads_started":1}
I20260927 05:06:40.462390 7736 tablet_service.cc:1467] Processing CreateTablet for tablet c6d562fde4374b76bb65cb81262b0c1e (DEFAULT_TABLE table=test-table [id=92276af5269b447280f89d8324c07266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:40.462926 7736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d562fde4374b76bb65cb81262b0c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:40.463721 7909 tablet_bootstrap.cc:492] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8: Bootstrap starting.
I20260927 05:06:40.464380 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.444572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.0.97.2:33669)
0927 05:06:40.457598 (+ 13026us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:40.457605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:40.457626 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:06:40.457696 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:40.457703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:40.457778 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:40.457896 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:40.458491 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.459500 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.461411 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.462686 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.462832 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:40.462950 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:40.463875 (+ 925us) server_negotiation.cc:299] Negotiation successful
0927 05:06:40.464082 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12871,"thread_start_us":13440,"threads_started":1}
I20260927 05:06:40.464597 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.444165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33669 (local address 127.0.0.1:37900)
0927 05:06:40.445221 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.445240 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:40.445341 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:40.457940 (+ 12599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:40.457947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:40.457979 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:40.458298 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:40.458310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.459658 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.459664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:40.461221 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:40.461234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.462931 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.462939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:40.463119 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:40.463155 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:06:40.463254 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 05:06:40.463403 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":385,"threads_started":1}
I20260927 05:06:40.465443 7909 tablet_bootstrap.cc:654] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:40.467661 7909 tablet_bootstrap.cc:492] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8: No bootstrap required, opened a new log
I20260927 05:06:40.467778 7909 ts_tablet_manager.cc:1397] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 05:06:40.468777 7909 raft_consensus.cc:348] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } } peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } }
I20260927 05:06:40.469486 7911 tablet_bootstrap.cc:492] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff: Bootstrap starting.
I20260927 05:06:40.469666 7595 tablet_service.cc:1467] Processing CreateTablet for tablet c6d562fde4374b76bb65cb81262b0c1e (DEFAULT_TABLE table=test-table [id=92276af5269b447280f89d8324c07266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:40.470163 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d562fde4374b76bb65cb81262b0c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:40.471323 7911 tablet_bootstrap.cc:654] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:40.472177 7909 raft_consensus.cc:374] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:40.472318 7909 raft_consensus.cc:729] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4f8eee7df847efb91934b3963549e8, State: Initialized, Role: FOLLOWER
I20260927 05:06:40.472601 7909 consensus_queue.cc:260] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } } peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } }
I20260927 05:06:40.473507 7909 ts_tablet_manager.cc:1428] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260927 05:06:40.480010 7516 heartbeater.cc:503] Master 127.0.97.61:38783 was elected leader, sending a full tablet report...
I20260927 05:06:40.480911 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.474297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.1:44883)
0927 05:06:40.480408 (+ 6111us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.480590 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5964,"threads_started":1}
I20260927 05:06:40.482412 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.481764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.1:53301)
0927 05:06:40.482063 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.482194 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260927 05:06:40.484148 7911 tablet_bootstrap.cc:492] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff: No bootstrap required, opened a new log
I20260927 05:06:40.484228 7915 tablet_bootstrap.cc:492] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678: Bootstrap starting.
I20260927 05:06:40.484426 7911 ts_tablet_manager.cc:1397] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 05:06:40.485222 7911 raft_consensus.cc:348] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } } peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } }
I20260927 05:06:40.485495 7911 raft_consensus.cc:374] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:40.485636 7911 raft_consensus.cc:729] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5a2a91d9a5430a9138d357ee7c1cff, State: Initialized, Role: FOLLOWER
I20260927 05:06:40.485929 7911 consensus_queue.cc:260] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } } peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } }
I20260927 05:06:40.486052 7915 tablet_bootstrap.cc:654] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:40.486819 7911 ts_tablet_manager.cc:1428] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 05:06:40.489008 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.487190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.3:56401)
0927 05:06:40.487865 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.488023 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":368,"threads_started":1}
I20260927 05:06:40.490464 7803 heartbeater.cc:503] Master 127.0.97.61:38783 was elected leader, sending a full tablet report...
I20260927 05:06:40.490960 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.489892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.3:54473)
0927 05:06:40.490460 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.490664 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":421,"threads_started":1}
I20260927 05:06:40.492295 7915 tablet_bootstrap.cc:492] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678: No bootstrap required, opened a new log
I20260927 05:06:40.492564 7915 ts_tablet_manager.cc:1397] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 05:06:40.493314 7915 raft_consensus.cc:348] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } } peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } }
I20260927 05:06:40.493511 7915 raft_consensus.cc:374] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:40.493573 7915 raft_consensus.cc:729] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1691dc7882c1481eb19f193d542bb678, State: Initialized, Role: FOLLOWER
I20260927 05:06:40.493824 7915 consensus_queue.cc:260] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } } peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } }
I20260927 05:06:40.495465 7915 ts_tablet_manager.cc:1428] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 05:06:40.496183 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.494910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.2:59703)
0927 05:06:40.495602 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.495753 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":343,"threads_started":1}
I20260927 05:06:40.496886 7662 heartbeater.cc:503] Master 127.0.97.61:38783 was elected leader, sending a full tablet report...
I20260927 05:06:40.497484 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.496910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.2:39355)
0927 05:06:40.497238 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.497328 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260927 05:06:40.552839 7913 raft_consensus.cc:482] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:40.553146 7913 raft_consensus.cc:504] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } } peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } }
I20260927 05:06:40.554183 7913 leader_election.cc:288] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1691dc7882c1481eb19f193d542bb678 (127.0.97.2:33669), 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097)
I20260927 05:06:40.554876 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d562fde4374b76bb65cb81262b0c1e" candidate_uuid: "ec4f8eee7df847efb91934b3963549e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1691dc7882c1481eb19f193d542bb678" is_pre_election: true
I20260927 05:06:40.555064 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d562fde4374b76bb65cb81262b0c1e" candidate_uuid: "ec4f8eee7df847efb91934b3963549e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" is_pre_election: true
I20260927 05:06:40.555186 7615 raft_consensus.cc:2454] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4f8eee7df847efb91934b3963549e8 in term 0.
I20260927 05:06:40.555368 7756 raft_consensus.cc:2454] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4f8eee7df847efb91934b3963549e8 in term 0.
I20260927 05:06:40.555845 7402 leader_election.cc:302] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1691dc7882c1481eb19f193d542bb678, ec4f8eee7df847efb91934b3963549e8; no voters:
I20260927 05:06:40.556222 7913 raft_consensus.cc:2781] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:06:40.556337 7913 raft_consensus.cc:482] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:06:40.556489 7913 raft_consensus.cc:3037] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:40.558746 7913 raft_consensus.cc:504] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } } peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } }
I20260927 05:06:40.559355 7913 leader_election.cc:288] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [CANDIDATE]: Term 1 election: Requested vote from peers 1691dc7882c1481eb19f193d542bb678 (127.0.97.2:33669), 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097)
I20260927 05:06:40.560096 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d562fde4374b76bb65cb81262b0c1e" candidate_uuid: "ec4f8eee7df847efb91934b3963549e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1691dc7882c1481eb19f193d542bb678"
I20260927 05:06:40.560164 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d562fde4374b76bb65cb81262b0c1e" candidate_uuid: "ec4f8eee7df847efb91934b3963549e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff"
I20260927 05:06:40.560321 7615 raft_consensus.cc:3037] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:40.560434 7756 raft_consensus.cc:3037] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:40.562479 7615 raft_consensus.cc:2454] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4f8eee7df847efb91934b3963549e8 in term 1.
I20260927 05:06:40.562557 7756 raft_consensus.cc:2454] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4f8eee7df847efb91934b3963549e8 in term 1.
I20260927 05:06:40.562947 7402 leader_election.cc:302] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1691dc7882c1481eb19f193d542bb678, ec4f8eee7df847efb91934b3963549e8; no voters:
I20260927 05:06:40.563364 7913 raft_consensus.cc:2781] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:40.564064 7913 raft_consensus.cc:686] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 1 LEADER]: Becoming Leader. State: Replica: ec4f8eee7df847efb91934b3963549e8, State: Running, Role: LEADER
I20260927 05:06:40.564427 7913 consensus_queue.cc:237] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } } peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } }
I20260927 05:06:40.565315 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.564882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.1:36827)
0927 05:06:40.565056 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.565142 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 05:06:40.568053 7147 catalog_manager.cc:5495] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec4f8eee7df847efb91934b3963549e8 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ec4f8eee7df847efb91934b3963549e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4f8eee7df847efb91934b3963549e8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 } health_report { overall_health: UNKNOWN } } }
I20260927 05:06:40.591249 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.585942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.0.97.1:45703)
0927 05:06:40.586187 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:40.586194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:40.586215 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:06:40.586458 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:40.586471 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:40.586544 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:40.586652 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:06:40.587045 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.588068 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.589422 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.589867 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.590012 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:40.590437 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:06:40.590555 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:06:40.590657 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:40.590814 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 05:06:40.590894 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 05:06:40.594949 7615 raft_consensus.cc:1261] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678 [term 1 FOLLOWER]: Refusing update from remote peer ec4f8eee7df847efb91934b3963549e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:40.594967 7756 raft_consensus.cc:1261] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff [term 1 FOLLOWER]: Refusing update from remote peer ec4f8eee7df847efb91934b3963549e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:40.595733 7922 consensus_queue.cc:1035] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d5a2a91d9a5430a9138d357ee7c1cff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:40.596174 7913 consensus_queue.cc:1035] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1691dc7882c1481eb19f193d542bb678" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:40.598752 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.598084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.2:46931)
0927 05:06:40.598379 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.598501 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260927 05:06:40.599324 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.598780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.3:39437)
0927 05:06:40.599043 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.599153 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260927 05:06:40.610666 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.610001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.2:40263)
0927 05:06:40.610353 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.610475 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239}
I20260927 05:06:40.613117 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.612352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.1:51643)
0927 05:06:40.612596 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.612742 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 05:06:40.617084 7927 mvcc.cc:205] Tried to move back new op lower bound from 7333829020029575168 to 7333829019915190272. Current Snapshot: MvccSnapshot[applied={T|T < 7333829020029575168}]
I20260927 05:06:40.619185 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.618538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.3:35063)
0927 05:06:40.618823 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.618976 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 05:06:40.620738 7147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:40.621050 7147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:40.622459 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.621900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.97.1:53249)
0927 05:06:40.622151 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.622250 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 05:06:40.631502 7147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd7692436bd94dcc91d8823bb2692aba: Sending DeleteTablet for 3 replicas of tablet c6d562fde4374b76bb65cb81262b0c1e
I20260927 05:06:40.632745 7736 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d562fde4374b76bb65cb81262b0c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:40 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46250
I20260927 05:06:40.633183 7449 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d562fde4374b76bb65cb81262b0c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:40 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60970
I20260927 05:06:40.633266 7735 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d562fde4374b76bb65cb81262b0c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:40 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46250
I20260927 05:06:40.633716 7448 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d562fde4374b76bb65cb81262b0c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:40 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60970
I20260927 05:06:40.633913 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d562fde4374b76bb65cb81262b0c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:40 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:37900
I20260927 05:06:40.634550 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 7323
I20260927 05:06:40.654515 7935 tablet_replica.cc:330] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff: stopping tablet replica
I20260927 05:06:40.654984 7935 raft_consensus.cc:2229] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:40.655578 7935 raft_consensus.cc:2258] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:40.656591 7937 tablet_replica.cc:330] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678: stopping tablet replica
I20260927 05:06:40.656843 7937 raft_consensus.cc:2229] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:40.657207 7937 raft_consensus.cc:2258] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:40.657413 7935 ts_tablet_manager.cc:1905] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:40.657929 7936 tablet_replica.cc:330] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8: stopping tablet replica
I20260927 05:06:40.658142 7936 raft_consensus.cc:2229] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:06:40.658578 7936 raft_consensus.cc:2258] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:40.659478 7937 ts_tablet_manager.cc:1905] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:40.663848 7936 ts_tablet_manager.cc:1905] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:40.665143 7935 ts_tablet_manager.cc:1918] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:40.665292 7935 log.cc:1196] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/c6d562fde4374b76bb65cb81262b0c1e
I20260927 05:06:40.665779 7935 ts_tablet_manager.cc:1939] T c6d562fde4374b76bb65cb81262b0c1e P 4d5a2a91d9a5430a9138d357ee7c1cff: Deleting consensus metadata
I20260927 05:06:40.666616 7937 ts_tablet_manager.cc:1918] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:40.666765 7937 log.cc:1196] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/c6d562fde4374b76bb65cb81262b0c1e
I20260927 05:06:40.667092 7132 catalog_manager.cc:4841] TS 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097): tablet c6d562fde4374b76bb65cb81262b0c1e (table test-table [id=92276af5269b447280f89d8324c07266]) successfully deleted
I20260927 05:06:40.667281 7937 ts_tablet_manager.cc:1939] T c6d562fde4374b76bb65cb81262b0c1e P 1691dc7882c1481eb19f193d542bb678: Deleting consensus metadata
W20260927 05:06:40.667851 7132 catalog_manager.cc:4808] TS 4d5a2a91d9a5430a9138d357ee7c1cff (127.0.97.3:40097): delete failed for tablet c6d562fde4374b76bb65cb81262b0c1e because the tablet was not found. No further retry: Not found: Tablet not found: c6d562fde4374b76bb65cb81262b0c1e
I20260927 05:06:40.668560 7132 catalog_manager.cc:4841] TS 1691dc7882c1481eb19f193d542bb678 (127.0.97.2:33669): tablet c6d562fde4374b76bb65cb81262b0c1e (table test-table [id=92276af5269b447280f89d8324c07266]) successfully deleted
I20260927 05:06:40.672452 7936 ts_tablet_manager.cc:1918] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:40.672628 7936 log.cc:1196] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/c6d562fde4374b76bb65cb81262b0c1e
I20260927 05:06:40.673169 7936 ts_tablet_manager.cc:1939] T c6d562fde4374b76bb65cb81262b0c1e P ec4f8eee7df847efb91934b3963549e8: Deleting consensus metadata
I20260927 05:06:40.674371 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.635968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.0.97.1:45703)
0927 05:06:40.636223 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:40.636233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:40.653689 (+ 17456us) server_negotiation.cc:407] Connection header received
0927 05:06:40.653949 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:40.653960 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:40.654095 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:40.654289 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:40.657183 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.658486 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.660595 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.661275 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.661417 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:40.665274 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:40.665300 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:40.665328 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:40.665377 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:40.670465 (+ 5088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:40.671401 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:40.671412 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:40.671416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:40.671615 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:40.672212 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:40.672218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:40.672222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:40.672818 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:40.673214 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:40.673598 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 05:06:40.673893 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:06:40.674541 7133 catalog_manager.cc:4841] TS ec4f8eee7df847efb91934b3963549e8 (127.0.97.1:45703): tablet c6d562fde4374b76bb65cb81262b0c1e (table test-table [id=92276af5269b447280f89d8324c07266]) successfully deleted
W20260927 05:06:40.675293 7133 catalog_manager.cc:4808] TS ec4f8eee7df847efb91934b3963549e8 (127.0.97.1:45703): delete failed for tablet c6d562fde4374b76bb65cb81262b0c1e because the tablet was not found. No further retry: Not found: Tablet not found: c6d562fde4374b76bb65cb81262b0c1e
I20260927 05:06:40.675699 7509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:40.801493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 7323
I20260927 05:06:40.824923 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 7526
I20260927 05:06:40.840545 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.825822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.0.97.2:33669)
0927 05:06:40.826045 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:40.826053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:40.826112 (+ 59us) server_negotiation.cc:407] Connection header received
0927 05:06:40.826364 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:40.826368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:40.826435 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:40.826563 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:40.828027 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.828787 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.830270 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.830770 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.830927 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:40.833841 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:40.833858 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:40.833876 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:40.833914 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:40.837054 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:40.837913 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:40.837923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:40.837926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:40.838082 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:40.838498 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:40.838504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:40.838507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:40.838903 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:40.839249 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:40.839807 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 05:06:40.840188 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:06:40.841315 7655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:40.867290 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.866761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51600)
0927 05:06:40.866981 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.867107 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 05:06:40.868671 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.868183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46281 (local address 127.0.0.1:51614)
0927 05:06:40.868368 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:40.868471 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 05:06:40.943079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 7526
I20260927 05:06:40.962105 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 7673
I20260927 05:06:40.975885 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:40.963118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.0.97.3:40097)
0927 05:06:40.963648 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:40.963652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:40.963670 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:06:40.963728 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:40.963731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:40.963799 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:40.963896 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:40.965279 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.965989 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.967204 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:40.967627 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:40.967726 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:40.970343 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:40.970358 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:40.970377 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:40.970420 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:40.973314 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:40.974002 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:40.974010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:40.974012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:40.974161 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:40.974492 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:40.974495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:40.974497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:40.974842 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:40.975093 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:40.975407 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 05:06:40.975603 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":315,"threads_started":1}
I20260927 05:06:40.976734 7796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:41.085883 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 7673
I20260927 05:06:41.104209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 7012
I20260927 05:06:41.105470 7178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:41.234591 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 7012
I20260927 05:06:41.257220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 7112
I20260927 05:06:41.258515 7366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:41.376959 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 7112
2026-09-27T05:06:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11690 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:06:41 dist-test-slave-hgs0 krb5kdc[7947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:06:41 dist-test-slave-hgs0 krb5kdc[7947](info): set up 2 sockets
Sep 27 05:06:41 dist-test-slave-hgs0 krb5kdc[7947](info): commencing operation
krb5kdc: starting...
W20260927 05:06:42.525390 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:06:42 dist-test-slave-hgs0 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260927 05:06:42.765247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38825
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:43.128582 7960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:43.128932 7960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:43.129104 7960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:43.137193 7960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:43.137322 7960 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 05:06:43.137434 7960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:43.137574 7960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:43.137647 7960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:43.137744 7960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:43.148260 7960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38825
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:43.149691 7960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:43.151185 7960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:43.163604 7965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:43.163638 7968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:43.163659 7966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:43.164474 7960 server_base.cc:1034] running on GCE node
I20260927 05:06:43.165699 7960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:43.167289 7960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:43.168576 7960 hybrid_clock.cc:648] HybridClock initialized: now 1790485603168488 us; error 50 us; skew 500 ppm
I20260927 05:06:43.168995 7960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:43 dist-test-slave-hgs0 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:43.172439 7960 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260927 05:06:43.174073 7960 webserver.cc:460] Webserver started at http://127.0.97.62:34567/ using document root <none> and password file <none>
I20260927 05:06:43.174861 7960 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:43.175016 7960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:43.175371 7960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:43.178675 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "f3c208689fd740ce859c98ec4fa324f1"
format_stamp: "Formatted at 2026-09-27 05:06:43 on dist-test-slave-hgs0"
server_key: "2feba16d15d4f60b7e0348320246a161"
server_key_iv: "e11d70e837594f6fb1e9fe9725dc7318"
server_key_version: "encryptionkey@0"
I20260927 05:06:43.179595 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "f3c208689fd740ce859c98ec4fa324f1"
format_stamp: "Formatted at 2026-09-27 05:06:43 on dist-test-slave-hgs0"
server_key: "2feba16d15d4f60b7e0348320246a161"
server_key_iv: "e11d70e837594f6fb1e9fe9725dc7318"
server_key_version: "encryptionkey@0"
I20260927 05:06:43.185689 7960 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260927 05:06:43.189909 7975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:43.191394 7960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 05:06:43.191608 7960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "f3c208689fd740ce859c98ec4fa324f1"
format_stamp: "Formatted at 2026-09-27 05:06:43 on dist-test-slave-hgs0"
server_key: "2feba16d15d4f60b7e0348320246a161"
server_key_iv: "e11d70e837594f6fb1e9fe9725dc7318"
server_key_version: "encryptionkey@0"
I20260927 05:06:43.191855 7960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:43.221386 7960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:43.226728 7960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:43.227105 7960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:43.252668 7960 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38825
I20260927 05:06:43.252688 8036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38825 every 8 connection(s)
I20260927 05:06:43.254277 7960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:43.258960 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:43.259505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 7960
I20260927 05:06:43.259733 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:43.260599 388 external_mini_cluster.cc:1428] Setting key 05c18b473ffedc2154296218286c8b4b
I20260927 05:06:43.275255 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1: Bootstrap starting.
Sep 27 05:06:43 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:43.279910 8037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:43.281390 8037 log.cc:824] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:43.285241 8037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1: No bootstrap required, opened a new log
I20260927 05:06:43.288800 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:43.263042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.0.97.62:38825)
0927 05:06:43.264118 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:43.264134 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:43.264190 (+ 56us) server_negotiation.cc:407] Connection header received
0927 05:06:43.266158 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:43.266188 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:43.266700 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:43.267118 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:43.269157 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:43.270669 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:43.272765 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:43.273717 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:43.273919 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:43.279434 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:43.279445 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:43.279464 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:43.279515 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:43.283878 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:43.284910 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:43.284918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:43.284926 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:43.285127 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:43.285608 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:43.285612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:43.285615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:43.286222 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:43.286503 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:43.286985 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 05:06:43.287509 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":522,"threads_started":1}
I20260927 05:06:43.294189 8037 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c208689fd740ce859c98ec4fa324f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38825 } }
I20260927 05:06:43.294605 8037 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:43.294731 8037 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c208689fd740ce859c98ec4fa324f1, State: Initialized, Role: FOLLOWER
I20260927 05:06:43.295434 8037 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c208689fd740ce859c98ec4fa324f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38825 } }
I20260927 05:06:43.295677 8037 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:43.295826 8037 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:43.296041 8037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:43.298744 8037 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c208689fd740ce859c98ec4fa324f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38825 } }
I20260927 05:06:43.299335 8037 leader_election.cc:302] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c208689fd740ce859c98ec4fa324f1; no voters:
I20260927 05:06:43.300088 8037 leader_election.cc:288] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:43.300277 8042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:43.301818 8042 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [term 1 LEADER]: Becoming Leader. State: Replica: f3c208689fd740ce859c98ec4fa324f1, State: Running, Role: LEADER
I20260927 05:06:43.302451 8042 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c208689fd740ce859c98ec4fa324f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38825 } }
I20260927 05:06:43.302593 8037 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:43.306351 8043 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3c208689fd740ce859c98ec4fa324f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c208689fd740ce859c98ec4fa324f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38825 } } }
I20260927 05:06:43.307004 8043 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:43.306468 8044 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3c208689fd740ce859c98ec4fa324f1. Latest consensus state: current_term: 1 leader_uuid: "f3c208689fd740ce859c98ec4fa324f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c208689fd740ce859c98ec4fa324f1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38825 } } }
I20260927 05:06:43.307541 8044 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:43.312330 8051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:43.319327 8051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:43.328634 8051 catalog_manager.cc:1347] Generated new cluster ID: 2721bb52ca794c40938da2430591cfec
I20260927 05:06:43.328794 8051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:43.348354 8051 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:43.350517 8051 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:43.364305 8051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1: Generated new TSK 0
I20260927 05:06:43.365617 8051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260927 05:06:43.424252 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38825
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:43.796801 8063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:43.797209 8063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:43.797322 8063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:43.805871 8063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:43.805999 8063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:43.806089 8063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:06:43.806175 8063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:43.817638 8063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:43.819010 8063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:43.820464 8063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:43.835251 8069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:43.835206 8071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:43.835306 8068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:43.837637 8063 server_base.cc:1034] running on GCE node
I20260927 05:06:43.838390 8063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:43.839535 8063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:43.840816 8063 hybrid_clock.cc:648] HybridClock initialized: now 1790485603840738 us; error 60 us; skew 500 ppm
I20260927 05:06:43.841270 8063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:43 dist-test-slave-hgs0 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:43.845157 8063 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 05:06:43.847075 8063 webserver.cc:460] Webserver started at http://127.0.97.1:45927/ using document root <none> and password file <none>
I20260927 05:06:43.848094 8063 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:43.848263 8063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:43.848683 8063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:43.852391 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "830d70f5a7824fd49de2691527de274d"
format_stamp: "Formatted at 2026-09-27 05:06:43 on dist-test-slave-hgs0"
server_key: "4ce478697efb83818035a34e514620cb"
server_key_iv: "01b481b001c93eb31a5cd97df1c2266b"
server_key_version: "encryptionkey@0"
I20260927 05:06:43.853408 8063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "830d70f5a7824fd49de2691527de274d"
format_stamp: "Formatted at 2026-09-27 05:06:43 on dist-test-slave-hgs0"
server_key: "4ce478697efb83818035a34e514620cb"
server_key_iv: "01b481b001c93eb31a5cd97df1c2266b"
server_key_version: "encryptionkey@0"
I20260927 05:06:43.859490 8063 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 05:06:43.863642 8078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:43.865469 8063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:43.865675 8063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "830d70f5a7824fd49de2691527de274d"
format_stamp: "Formatted at 2026-09-27 05:06:43 on dist-test-slave-hgs0"
server_key: "4ce478697efb83818035a34e514620cb"
server_key_iv: "01b481b001c93eb31a5cd97df1c2266b"
server_key_version: "encryptionkey@0"
I20260927 05:06:43.865934 8063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:43.886413 8063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:43.891472 8063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:43.891790 8063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:43.894835 8063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:43.894951 8063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:43.895130 8063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:43.895223 8063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:06:43 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:43.924618 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:43.895426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49091 (local address 127.0.97.62:38825)
0927 05:06:43.896284 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:43.896289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:43.897401 (+ 1112us) server_negotiation.cc:407] Connection header received
0927 05:06:43.899670 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:43.899676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:43.899767 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:43.899912 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:43.902552 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:43.904062 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:43.907791 (+ 3729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:43.908570 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:43.908752 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:43.915176 (+ 6424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:43.915188 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:43.915197 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:43.915257 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:43.918710 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:43.920326 (+ 1616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:43.920335 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:43.920340 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:43.920542 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:43.921449 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:43.921457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:43.921461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:43.921952 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:43.922214 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:43.923740 (+ 1526us) server_negotiation.cc:299] Negotiation successful
0927 05:06:43.924073 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":562,"threads_started":1}
I20260927 05:06:43.927017 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:43.896249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38825 (local address 127.0.97.1:49091)
0927 05:06:43.897169 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:43.897254 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:43.899354 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:43.900292 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:43.900308 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:43.901256 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:43.902211 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:43.902270 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:43.904350 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:43.904363 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:43.907547 (+ 3184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:43.907580 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:43.908944 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:43.908955 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:43.909476 (+ 521us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:43.910551 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:43.910602 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:43.914719 (+ 4117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:43.918959 (+ 4240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:43.918980 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:43.919010 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:43.920107 (+ 1097us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:43.920742 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:43.920753 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:43.920761 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:43.921228 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:43.922237 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:43.922250 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:43.923044 (+ 794us) client_negotiation.cc:769] Sending connection context
0927 05:06:43.923584 (+ 540us) client_negotiation.cc:240] Negotiation successful
0927 05:06:43.924544 (+ 960us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":391,"threads_started":1}
I20260927 05:06:43.942072 8063 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36653
I20260927 05:06:43.942139 8197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36653 every 8 connection(s)
I20260927 05:06:43.943650 8063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:43.946725 8199 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38825
I20260927 05:06:43.948005 8199 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:43.948848 8199 heartbeater.cc:511] Master 127.0.97.62:38825 requested a full tablet report, sending...
I20260927 05:06:43.949098 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 8063
I20260927 05:06:43.949316 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:06:43.950207 388 external_mini_cluster.cc:1428] Setting key 66ce524354d1a9abaa1f89647b6c0ae1
I20260927 05:06:43.951581 7992 ts_manager.cc:194] Registered new tserver with Master: 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653)
I20260927 05:06:43.953640 7992 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49091
I20260927 05:06:43.960069 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:43.943251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59823 (local address 127.0.97.62:38825)
0927 05:06:43.943494 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:43.943503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:43.944300 (+ 797us) server_negotiation.cc:407] Connection header received
0927 05:06:43.944611 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:43.944617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:43.944722 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:43.944846 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:43.946419 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:43.947301 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:43.949544 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:43.950397 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:43.950553 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:43.953221 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:43.953228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:43.953234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:43.953273 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:43.956678 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:43.957690 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:43.957696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:43.957701 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:43.957858 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:43.958325 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:43.958338 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:43.958340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:43.958656 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:43.958835 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:43.959308 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 05:06:43.959590 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:06:43.960126 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:43.943268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38825 (local address 127.0.97.1:59823)
0927 05:06:43.944152 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:43.944177 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:43.944405 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:43.945072 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:43.945081 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:43.945851 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:43.946185 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:43.946223 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:43.947486 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:43.947494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:43.949310 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:43.949332 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:43.950831 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:43.950838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:43.951033 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:43.951813 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:43.951842 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:43.952991 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:43.956931 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:43.957001 (+ 70us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:43.957005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:43.957508 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:43.958019 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:43.958025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:43.958029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:43.958184 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:43.958792 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:43.958798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:43.958988 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 05:06:43.959192 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 05:06:43.959591 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":471,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260927 05:06:44.008461 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38825
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:44.386278 8203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:44.386674 8203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:44.386881 8203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:44.395040 8203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:44.395236 8203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:44.395427 8203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:06:44.395546 8203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:44.406203 8203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:44.407497 8203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:44.409017 8203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:44.422873 8208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:44.422833 8209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:44.422897 8211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:44.423928 8203 server_base.cc:1034] running on GCE node
I20260927 05:06:44.424587 8203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:44.425529 8203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:44.426789 8203 hybrid_clock.cc:648] HybridClock initialized: now 1790485604426673 us; error 88 us; skew 500 ppm
I20260927 05:06:44.427234 8203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:44 dist-test-slave-hgs0 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:44.430969 8203 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 05:06:44.432775 8203 webserver.cc:460] Webserver started at http://127.0.97.2:46381/ using document root <none> and password file <none>
I20260927 05:06:44.433602 8203 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:44.433749 8203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:44.434119 8203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:44.437422 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "83505fa980a94874afb597582cd1106f"
format_stamp: "Formatted at 2026-09-27 05:06:44 on dist-test-slave-hgs0"
server_key: "95dde7a1743293cec9e3275e5f321c02"
server_key_iv: "c87f8d379bdb7d644adebe91c37895a1"
server_key_version: "encryptionkey@0"
I20260927 05:06:44.438330 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "83505fa980a94874afb597582cd1106f"
format_stamp: "Formatted at 2026-09-27 05:06:44 on dist-test-slave-hgs0"
server_key: "95dde7a1743293cec9e3275e5f321c02"
server_key_iv: "c87f8d379bdb7d644adebe91c37895a1"
server_key_version: "encryptionkey@0"
I20260927 05:06:44.444208 8203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 05:06:44.447815 8218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:44.449316 8203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:44.449507 8203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "83505fa980a94874afb597582cd1106f"
format_stamp: "Formatted at 2026-09-27 05:06:44 on dist-test-slave-hgs0"
server_key: "95dde7a1743293cec9e3275e5f321c02"
server_key_iv: "c87f8d379bdb7d644adebe91c37895a1"
server_key_version: "encryptionkey@0"
I20260927 05:06:44.449749 8203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:44.462255 8203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:44.467808 8203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:44.468176 8203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:44.471300 8203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:44.471437 8203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:44.471583 8203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:44.471657 8203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:06:44 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:44.497664 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:44.472270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49849 (local address 127.0.97.62:38825)
0927 05:06:44.472865 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:44.472870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:44.473991 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 05:06:44.476133 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:44.476138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:44.476229 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:44.476360 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:44.478753 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:44.480242 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:44.481985 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:44.482896 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:44.483055 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:44.488027 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:44.488038 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:44.488047 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:44.488108 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:44.492208 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:44.493533 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:44.493540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:44.493544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:44.493707 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:44.494396 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:44.494405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:44.494409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:44.494833 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:44.495065 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:44.496699 (+ 1634us) server_negotiation.cc:299] Negotiation successful
0927 05:06:44.497054 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":364,"threads_started":1}
I20260927 05:06:44.501511 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:44.472993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38825 (local address 127.0.97.2:49849)
0927 05:06:44.473796 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:44.473846 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:44.475749 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:44.476667 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:44.476683 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:44.477580 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:44.478493 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:44.478551 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:44.480426 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:44.480431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:44.481817 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:44.481829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:44.483196 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:44.483209 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:44.483495 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:44.484516 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:44.484550 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:44.487673 (+ 3123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:44.492437 (+ 4764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:44.492453 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:44.492481 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:44.493332 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:44.493854 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:44.493865 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:44.493869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:44.494175 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:44.495047 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:44.495063 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:44.495867 (+ 804us) client_negotiation.cc:769] Sending connection context
0927 05:06:44.497996 (+ 2129us) client_negotiation.cc:240] Negotiation successful
0927 05:06:44.498954 (+ 958us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":331,"threads_started":1}
I20260927 05:06:44.527494 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:44.509613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44717 (local address 127.0.97.62:38825)
0927 05:06:44.509903 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:44.509914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:44.510962 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 05:06:44.511237 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:44.511247 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:44.511375 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:44.511560 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:44.513175 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:44.514580 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:44.516275 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:44.517025 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:44.517215 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:44.520319 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:44.520327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:44.520332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:44.520393 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:44.524316 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:44.525411 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:44.525415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:44.525417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:44.525561 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:44.526039 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:44.526042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:44.526044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:44.526371 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:44.526502 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:44.527062 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 05:06:44.527239 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 05:06:44.528223 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:44.509916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38825 (local address 127.0.97.2:44717)
0927 05:06:44.510777 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:44.510803 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:44.511054 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:44.511632 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:44.511637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:44.512266 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:44.512901 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:44.512988 (+ 87us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:44.514766 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:44.514773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:44.516085 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:44.516101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:44.517308 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:44.517317 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:44.517548 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:44.518313 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:44.518340 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:44.520035 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:44.524497 (+ 4462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:44.524506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:44.524514 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:44.525245 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:44.525695 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:44.525703 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:44.525710 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:44.525909 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:44.526503 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:44.526511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:44.526788 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 05:06:44.526969 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 05:06:44.527452 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":497,"threads_started":1}
I20260927 05:06:44.532661 8203 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43505
I20260927 05:06:44.532712 8338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43505 every 8 connection(s)
I20260927 05:06:44.534135 8203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:44.534713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 8203
I20260927 05:06:44.534970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:06:44.535542 388 external_mini_cluster.cc:1428] Setting key bff7cd8b5e18b9e4e3c90d7475183628
I20260927 05:06:44.536667 8339 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38825
I20260927 05:06:44.536942 8339 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:44.537813 8339 heartbeater.cc:511] Master 127.0.97.62:38825 requested a full tablet report, sending...
I20260927 05:06:44.539458 7992 ts_manager.cc:194] Registered new tserver with Master: 83505fa980a94874afb597582cd1106f (127.0.97.2:43505)
I20260927 05:06:44.540854 7992 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49849
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 05:06:44.588414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38825
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:44.933498 8343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:44.933794 8343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:44.933959 8343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:44.941599 8343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:44.941736 8343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:44.941859 8343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 05:06:44.941972 8343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:44.955209 8343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:44.956703 8343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:44.958112 8343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 05:06:44.958789 8199 heartbeater.cc:503] Master 127.0.97.62:38825 was elected leader, sending a full tablet report...
W20260927 05:06:44.973431 8348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:44.973551 8351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:44.973507 8349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:44.974735 8343 server_base.cc:1034] running on GCE node
I20260927 05:06:44.975447 8343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:44.976603 8343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:44.977842 8343 hybrid_clock.cc:648] HybridClock initialized: now 1790485604977758 us; error 75 us; skew 500 ppm
I20260927 05:06:44.978158 8343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:44 dist-test-slave-hgs0 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:44.981772 8343 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 05:06:44.983444 8343 webserver.cc:460] Webserver started at http://127.0.97.3:46225/ using document root <none> and password file <none>
I20260927 05:06:44.984305 8343 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:44.984431 8343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:44.984789 8343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:44.988055 8343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "c2cdeca93e41404597da9c0742b975c2"
format_stamp: "Formatted at 2026-09-27 05:06:44 on dist-test-slave-hgs0"
server_key: "866d6cb762cf93464181e5082e3a9f88"
server_key_iv: "898fde17df644e7b482c3c9d2d0783d7"
server_key_version: "encryptionkey@0"
I20260927 05:06:44.988960 8343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c2cdeca93e41404597da9c0742b975c2"
format_stamp: "Formatted at 2026-09-27 05:06:44 on dist-test-slave-hgs0"
server_key: "866d6cb762cf93464181e5082e3a9f88"
server_key_iv: "898fde17df644e7b482c3c9d2d0783d7"
server_key_version: "encryptionkey@0"
I20260927 05:06:44.995283 8343 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 05:06:44.999614 8358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:45.001369 8343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:45.001556 8343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "c2cdeca93e41404597da9c0742b975c2"
format_stamp: "Formatted at 2026-09-27 05:06:44 on dist-test-slave-hgs0"
server_key: "866d6cb762cf93464181e5082e3a9f88"
server_key_iv: "898fde17df644e7b482c3c9d2d0783d7"
server_key_version: "encryptionkey@0"
I20260927 05:06:45.001827 8343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:45.035866 8343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:45.040145 8343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:45.040431 8343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:45.043056 8343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:45.043174 8343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:45.043329 8343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:45.043406 8343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:06:45 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:45.073900 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.043893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44915 (local address 127.0.97.62:38825)
0927 05:06:45.046303 (+ 2410us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.046309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.046333 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:06:45.048135 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.048141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.048230 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.048363 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:45.050752 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.051840 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.055355 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.056386 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.056605 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.065111 (+ 8506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:45.065165 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:45.065171 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:45.065264 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:45.069138 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:45.070484 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:45.070493 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:45.070499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:45.070759 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:45.071271 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:45.071279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:45.071285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:45.071820 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:45.072034 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.072942 (+ 908us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.073297 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2276,"thread_start_us":2156,"threads_started":1}
I20260927 05:06:45.074525 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.044548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38825 (local address 127.0.97.3:44915)
0927 05:06:45.045413 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:45.045473 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:45.047764 (+ 2291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:45.048654 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:45.048663 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:45.049505 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:45.050542 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.050564 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.052112 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.052120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.054945 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.055026 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.056817 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.056835 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.057385 (+ 550us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.058705 (+ 1320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:45.058741 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:45.064639 (+ 5898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:45.069342 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:45.069357 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:45.069390 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:45.070301 (+ 911us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:45.070913 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:45.070917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:45.070919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:45.071149 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:45.072062 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:45.072071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:45.072477 (+ 406us) client_negotiation.cc:769] Sending connection context
0927 05:06:45.072736 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 05:06:45.073202 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":366,"threads_started":1}
I20260927 05:06:45.097479 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.081382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54741 (local address 127.0.97.62:38825)
0927 05:06:45.081596 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.081601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.082548 (+ 947us) server_negotiation.cc:407] Connection header received
0927 05:06:45.082787 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.082793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.082872 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.082991 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:45.084120 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.084914 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.086317 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.086750 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.086911 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.089949 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:45.089955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:45.089958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:45.090023 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:45.094032 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:45.095185 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:45.095194 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:45.095197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:45.095407 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:45.095864 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:45.095871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:45.095876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:45.096222 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:45.096421 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.096874 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.097108 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:06:45.097854 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.081487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38825 (local address 127.0.97.3:54741)
0927 05:06:45.082415 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:45.082434 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:45.082632 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:45.083021 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:45.083026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:45.083580 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:45.083894 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.083904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.085064 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.085072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.086163 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.086175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.086969 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.086975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.087151 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.088029 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:45.088049 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:45.089690 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:45.094333 (+ 4643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:45.094341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:45.094347 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:45.094970 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:45.095534 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:45.095560 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:45.095564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:45.095708 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:45.096378 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:45.096383 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:45.096579 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 05:06:45.096771 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 05:06:45.097174 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":510,"threads_started":1}
I20260927 05:06:45.110378 8343 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45093
I20260927 05:06:45.110487 8478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45093 every 8 connection(s)
I20260927 05:06:45.111711 8343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:45.113082 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 8343
I20260927 05:06:45.113273 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:06:45.113699 388 external_mini_cluster.cc:1428] Setting key ac47469d48e5b96c6babcf220410b5a2
I20260927 05:06:45.113866 8479 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38825
I20260927 05:06:45.114641 8479 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:45.115525 8479 heartbeater.cc:511] Master 127.0.97.62:38825 requested a full tablet report, sending...
I20260927 05:06:45.117024 7992 ts_manager.cc:194] Registered new tserver with Master: c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093)
I20260927 05:06:45.117810 7992 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44915
I20260927 05:06:45.129881 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 05:06:45 dist-test-slave-hgs0 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 05:06:45 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:45 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:06:45.183157 8367 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:45.171509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.0.97.62:38825)
0927 05:06:45.171702 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.171707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.172023 (+ 316us) server_negotiation.cc:407] Connection header received
0927 05:06:45.172236 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.172240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.172313 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.172417 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:45.173905 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.174822 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.176359 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.176819 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.176929 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.182862 (+ 5933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38540: BlockingRecv error: recv got EOF from 127.0.0.1:38540 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 05:06:45 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:45.200831 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.186632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.0.97.62:38825)
0927 05:06:45.186880 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.186887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.187181 (+ 294us) server_negotiation.cc:407] Connection header received
0927 05:06:45.187422 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.187427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.187501 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.187610 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:45.188926 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.189641 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.190965 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.191421 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.191579 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.195511 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:45.195516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:45.195520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:45.195588 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:45.198434 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:45.199086 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:45.199090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:45.199092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:45.199222 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:45.199544 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:45.199547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:45.199549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:45.199804 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:45.200100 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.200315 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.200487 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 05:06:45.205160 7992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:06:45.212792 7992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:06:45.241319 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.226284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.0.1:53002)
0927 05:06:45.229400 (+ 3116us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:45.229435 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:45.229768 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:45.230772 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:45.230785 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:45.230838 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:45.231449 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.231467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.235131 (+ 3664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.235137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.237953 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.237970 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.240208 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.240216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.240413 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.240486 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 05:06:45.240622 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 05:06:45.240841 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2911,"thread_start_us":2687,"threads_started":1}
I20260927 05:06:45.242935 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.226779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.0.97.1:36653)
0927 05:06:45.228725 (+ 1946us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.228739 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.229636 (+ 897us) server_negotiation.cc:407] Connection header received
0927 05:06:45.230065 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.230079 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.230451 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.231721 (+ 1270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:45.232055 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.234312 (+ 2257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.238177 (+ 3865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.239921 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.241946 (+ 2025us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.242207 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.242295 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.242629 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1714,"thread_start_us":1548,"threads_started":1}
I20260927 05:06:45.244155 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.226037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45093 (local address 127.0.0.1:41474)
0927 05:06:45.227112 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:45.227176 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:45.227433 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:45.235908 (+ 8475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:45.235914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:45.235943 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:45.236436 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.236450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.238834 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.238840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.241079 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.241094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.243198 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.243210 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.243456 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.243488 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:06:45.243595 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 05:06:45.243750 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":491,"threads_started":1}
I20260927 05:06:45.244627 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.226374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.0.97.3:45093)
0927 05:06:45.235160 (+ 8786us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.235172 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.235211 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:06:45.235322 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.235333 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.235650 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.236675 (+ 1025us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:45.237012 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.238661 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.241330 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.242905 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.243683 (+ 778us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.243970 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.244067 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.244259 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8570,"thread_start_us":8360,"threads_started":1}
I20260927 05:06:45.247874 8129 tablet_service.cc:1467] Processing CreateTablet for tablet 4be86875ab38427a90ce25745e9a2940 (DEFAULT_TABLE table=test-table [id=cac58269c8834416bf7d453c009e3245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:45.248194 8409 tablet_service.cc:1467] Processing CreateTablet for tablet 4be86875ab38427a90ce25745e9a2940 (DEFAULT_TABLE table=test-table [id=cac58269c8834416bf7d453c009e3245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:45.249560 8129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be86875ab38427a90ce25745e9a2940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:45.249763 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be86875ab38427a90ce25745e9a2940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:45.252899 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.229466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.0.97.2:43505)
0927 05:06:45.240952 (+ 11486us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.240968 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.242299 (+ 1331us) server_negotiation.cc:407] Connection header received
0927 05:06:45.242429 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.242438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.242711 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.242964 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:45.243645 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.245891 (+ 2246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.248497 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.250047 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.250205 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.252276 (+ 2071us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.252369 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.252568 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11197,"thread_start_us":12544,"threads_started":1}
I20260927 05:06:45.253242 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.228772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43505 (local address 127.0.0.1:43218)
0927 05:06:45.241519 (+ 12747us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:45.241538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:45.241683 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:45.242915 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:45.242920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:45.242952 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:45.243369 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.243383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.246087 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.246093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.248277 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.248296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.250804 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.250811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.251023 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.251065 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 05:06:45.251165 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 05:06:45.251318 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12610,"thread_start_us":12431,"threads_started":1}
I20260927 05:06:45.257783 8269 tablet_service.cc:1467] Processing CreateTablet for tablet 4be86875ab38427a90ce25745e9a2940 (DEFAULT_TABLE table=test-table [id=cac58269c8834416bf7d453c009e3245]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:45.259455 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be86875ab38427a90ce25745e9a2940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:45.275611 8503 tablet_bootstrap.cc:492] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: Bootstrap starting.
I20260927 05:06:45.279266 8505 tablet_bootstrap.cc:492] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: Bootstrap starting.
I20260927 05:06:45.279796 8503 tablet_bootstrap.cc:654] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:45.281313 8503 log.cc:824] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:45.283286 8505 tablet_bootstrap.cc:654] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:45.285393 8505 log.cc:824] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:45.286463 8504 tablet_bootstrap.cc:492] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: Bootstrap starting.
I20260927 05:06:45.290113 8503 tablet_bootstrap.cc:492] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: No bootstrap required, opened a new log
I20260927 05:06:45.290253 8505 tablet_bootstrap.cc:492] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: No bootstrap required, opened a new log
I20260927 05:06:45.290517 8503 ts_tablet_manager.cc:1397] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260927 05:06:45.290567 8505 ts_tablet_manager.cc:1397] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260927 05:06:45.291337 8504 tablet_bootstrap.cc:654] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:45.292708 8504 log.cc:824] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:45.296379 8504 tablet_bootstrap.cc:492] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: No bootstrap required, opened a new log
I20260927 05:06:45.296692 8504 ts_tablet_manager.cc:1397] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 05:06:45.298933 8503 raft_consensus.cc:348] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } }
I20260927 05:06:45.299376 8503 raft_consensus.cc:374] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:45.299525 8503 raft_consensus.cc:729] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d70f5a7824fd49de2691527de274d, State: Initialized, Role: FOLLOWER
I20260927 05:06:45.300379 8503 consensus_queue.cc:260] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } }
I20260927 05:06:45.301616 8505 raft_consensus.cc:348] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } }
I20260927 05:06:45.302136 8505 raft_consensus.cc:374] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:45.302275 8505 raft_consensus.cc:729] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83505fa980a94874afb597582cd1106f, State: Initialized, Role: FOLLOWER
I20260927 05:06:45.302369 8503 ts_tablet_manager.cc:1428] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s
I20260927 05:06:45.302954 8505 consensus_queue.cc:260] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } }
I20260927 05:06:45.305346 8339 heartbeater.cc:503] Master 127.0.97.62:38825 was elected leader, sending a full tablet report...
I20260927 05:06:45.306067 8505 ts_tablet_manager.cc:1428] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260927 05:06:45.308866 8504 raft_consensus.cc:348] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } }
I20260927 05:06:45.309371 8504 raft_consensus.cc:374] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:45.309489 8504 raft_consensus.cc:729] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cdeca93e41404597da9c0742b975c2, State: Initialized, Role: FOLLOWER
I20260927 05:06:45.310266 8504 consensus_queue.cc:260] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } }
I20260927 05:06:45.311354 8479 heartbeater.cc:503] Master 127.0.97.62:38825 was elected leader, sending a full tablet report...
I20260927 05:06:45.312121 8504 ts_tablet_manager.cc:1428] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
W20260927 05:06:45.363425 8480 tablet.cc:2369] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:06:45.445515 8510 raft_consensus.cc:482] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:45.445899 8510 raft_consensus.cc:504] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } }
W20260927 05:06:45.446990 8200 tablet.cc:2369] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:06:45.448114 8510 leader_election.cc:288] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093), 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653)
I20260927 05:06:45.455433 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.448625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50453 (local address 127.0.97.1:36653)
0927 05:06:45.448853 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.448862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.449727 (+ 865us) server_negotiation.cc:407] Connection header received
0927 05:06:45.449919 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.449923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.450023 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.450114 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:45.450744 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.451419 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.453471 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.454336 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.454445 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.454561 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.455005 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.455156 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:06:45.455502 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.448552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43507 (local address 127.0.97.3:45093)
0927 05:06:45.448809 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.448819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.449586 (+ 767us) server_negotiation.cc:407] Connection header received
0927 05:06:45.449842 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.449847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.449927 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.449999 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:45.450703 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.451420 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.453496 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.454471 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.454603 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.454710 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.455104 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.455233 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:06:45.455614 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.448402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45093 (local address 127.0.97.2:43507)
0927 05:06:45.449444 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:45.449460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:45.449657 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:45.450041 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:45.450048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:45.450078 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:45.450552 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.450562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.451591 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.451594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.453299 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.453310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.454760 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.454768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.454919 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.454959 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 05:06:45.455063 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:06:45.455203 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":149,"spinlock_wait_cycles":51968,"thread_start_us":628,"threads_started":1}
I20260927 05:06:45.456103 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.448516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.97.2:50453)
0927 05:06:45.449618 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:45.449635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:45.449770 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:45.450160 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:45.450163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:45.450195 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:45.450563 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.450571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.451555 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.451561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.453288 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.453300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.454588 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.454595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.454811 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.454836 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 05:06:45.454974 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 05:06:45.455155 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":29,"spinlock_wait_cycles":29440,"thread_start_us":771,"threads_started":1}
I20260927 05:06:45.456758 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be86875ab38427a90ce25745e9a2940" candidate_uuid: "83505fa980a94874afb597582cd1106f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2cdeca93e41404597da9c0742b975c2" is_pre_election: true
I20260927 05:06:45.457149 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be86875ab38427a90ce25745e9a2940" candidate_uuid: "83505fa980a94874afb597582cd1106f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d70f5a7824fd49de2691527de274d" is_pre_election: true
I20260927 05:06:45.457397 8434 raft_consensus.cc:2454] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83505fa980a94874afb597582cd1106f in term 0.
I20260927 05:06:45.457664 8149 raft_consensus.cc:2454] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83505fa980a94874afb597582cd1106f in term 0.
I20260927 05:06:45.458233 8222 leader_election.cc:302] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 830d70f5a7824fd49de2691527de274d, 83505fa980a94874afb597582cd1106f, c2cdeca93e41404597da9c0742b975c2; no voters:
I20260927 05:06:45.458745 8510 raft_consensus.cc:2781] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:06:45.458909 8510 raft_consensus.cc:482] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:06:45.459072 8510 raft_consensus.cc:3037] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:45.461433 8510 raft_consensus.cc:504] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } }
I20260927 05:06:45.462083 8510 leader_election.cc:288] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [CANDIDATE]: Term 1 election: Requested vote from peers c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093), 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653)
I20260927 05:06:45.462697 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be86875ab38427a90ce25745e9a2940" candidate_uuid: "83505fa980a94874afb597582cd1106f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d70f5a7824fd49de2691527de274d"
I20260927 05:06:45.462697 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be86875ab38427a90ce25745e9a2940" candidate_uuid: "83505fa980a94874afb597582cd1106f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2cdeca93e41404597da9c0742b975c2"
I20260927 05:06:45.462925 8434 raft_consensus.cc:3037] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:45.462925 8149 raft_consensus.cc:3037] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:45.466466 8149 raft_consensus.cc:2454] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83505fa980a94874afb597582cd1106f in term 1.
I20260927 05:06:45.466475 8434 raft_consensus.cc:2454] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83505fa980a94874afb597582cd1106f in term 1.
I20260927 05:06:45.467043 8222 leader_election.cc:302] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 830d70f5a7824fd49de2691527de274d, 83505fa980a94874afb597582cd1106f, c2cdeca93e41404597da9c0742b975c2; no voters:
I20260927 05:06:45.467398 8510 raft_consensus.cc:2781] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:45.467986 8510 raft_consensus.cc:686] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 1 LEADER]: Becoming Leader. State: Replica: 83505fa980a94874afb597582cd1106f, State: Running, Role: LEADER
I20260927 05:06:45.468715 8510 consensus_queue.cc:237] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } }
I20260927 05:06:45.474527 7991 catalog_manager.cc:5495] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f reported cstate change: term changed from 0 to 1, leader changed from <none> to 83505fa980a94874afb597582cd1106f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "83505fa980a94874afb597582cd1106f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } health_report { overall_health: HEALTHY } } }
I20260927 05:06:45.490027 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.484925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.0.97.2:43505)
0927 05:06:45.485142 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.485147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.485201 (+ 54us) server_negotiation.cc:407] Connection header received
0927 05:06:45.485408 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.485412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.485497 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.485638 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:06:45.485918 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.486625 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.487889 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.488366 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.488472 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.488856 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:06:45.489403 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:06:45.489522 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.489659 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.489726 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:06:45.498700 8434 raft_consensus.cc:1261] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Refusing update from remote peer 83505fa980a94874afb597582cd1106f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:45.498955 8149 raft_consensus.cc:1261] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Refusing update from remote peer 83505fa980a94874afb597582cd1106f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:45.499866 8514 consensus_queue.cc:1035] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:45.500303 8510 consensus_queue.cc:1035] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:45.518889 8518 mvcc.cc:205] Tried to move back new op lower bound from 7333829040112156672 to 7333829040005611520. Current Snapshot: MvccSnapshot[applied={T|T < 7333829040112156672}]
I20260927 05:06:45.536183 7991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:45.536520 7991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:45.541738 7991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1: Sending DeleteTablet for 3 replicas of tablet 4be86875ab38427a90ce25745e9a2940
I20260927 05:06:45.543743 8409 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be86875ab38427a90ce25745e9a2940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41474
I20260927 05:06:45.543768 8408 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be86875ab38427a90ce25745e9a2940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41474
I20260927 05:06:45.544850 8527 tablet_replica.cc:330] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: stopping tablet replica
I20260927 05:06:45.545328 8527 raft_consensus.cc:2229] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:45.545938 8527 raft_consensus.cc:2258] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:45.545945 8129 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be86875ab38427a90ce25745e9a2940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53002
I20260927 05:06:45.547652 7992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 05:06:45.548324 8527 ts_tablet_manager.cc:1905] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 05:06:45.548450 7992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:06:45.552220 8269 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be86875ab38427a90ce25745e9a2940 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43218
I20260927 05:06:45.553779 8528 tablet_replica.cc:330] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: stopping tablet replica
I20260927 05:06:45.554276 8528 raft_consensus.cc:2229] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:45.554900 8528 raft_consensus.cc:2258] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:45.559674 8129 tablet_service.cc:1467] Processing CreateTablet for tablet cb764202b3a5425fa2bdff8ef7e2d057 (DEFAULT_TABLE table=test-table [id=6a9cb8d084ed47efb86c57fb516b8976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:45.560246 8129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb764202b3a5425fa2bdff8ef7e2d057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:45.565097 8409 tablet_service.cc:1467] Processing CreateTablet for tablet cb764202b3a5425fa2bdff8ef7e2d057 (DEFAULT_TABLE table=test-table [id=6a9cb8d084ed47efb86c57fb516b8976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:45.565663 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb764202b3a5425fa2bdff8ef7e2d057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:45.565630 8528 ts_tablet_manager.cc:1905] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:45.566272 8529 tablet_replica.cc:330] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: stopping tablet replica
I20260927 05:06:45.566843 8529 raft_consensus.cc:2229] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:06:45.567528 8529 raft_consensus.cc:2258] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:45.571257 8503 tablet_bootstrap.cc:492] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d: Bootstrap starting.
I20260927 05:06:45.571795 8269 tablet_service.cc:1467] Processing CreateTablet for tablet cb764202b3a5425fa2bdff8ef7e2d057 (DEFAULT_TABLE table=test-table [id=6a9cb8d084ed47efb86c57fb516b8976]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:45.572249 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb764202b3a5425fa2bdff8ef7e2d057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:45.573436 8528 ts_tablet_manager.cc:1918] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:45.573504 8503 tablet_bootstrap.cc:654] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:45.573638 8528 log.cc:1196] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/4be86875ab38427a90ce25745e9a2940
I20260927 05:06:45.574265 8528 ts_tablet_manager.cc:1939] T 4be86875ab38427a90ce25745e9a2940 P 830d70f5a7824fd49de2691527de274d: Deleting consensus metadata
I20260927 05:06:45.576807 8527 ts_tablet_manager.cc:1918] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:45.577986 7977 catalog_manager.cc:4841] TS 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653): tablet 4be86875ab38427a90ce25745e9a2940 (table test-table [id=cac58269c8834416bf7d453c009e3245]) successfully deleted
I20260927 05:06:45.578255 8505 tablet_bootstrap.cc:492] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: Bootstrap starting.
I20260927 05:06:45.580195 8505 tablet_bootstrap.cc:654] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:45.579764 8504 tablet_bootstrap.cc:492] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2: Bootstrap starting.
I20260927 05:06:45.580547 8527 log.cc:1196] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/4be86875ab38427a90ce25745e9a2940
I20260927 05:06:45.581239 8527 ts_tablet_manager.cc:1939] T 4be86875ab38427a90ce25745e9a2940 P c2cdeca93e41404597da9c0742b975c2: Deleting consensus metadata
I20260927 05:06:45.582221 8504 tablet_bootstrap.cc:654] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:45.583487 8503 tablet_bootstrap.cc:492] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d: No bootstrap required, opened a new log
I20260927 05:06:45.583717 8503 ts_tablet_manager.cc:1397] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20260927 05:06:45.584769 8503 raft_consensus.cc:348] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.585085 8503 raft_consensus.cc:374] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:45.585124 7979 catalog_manager.cc:4841] TS c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093): tablet 4be86875ab38427a90ce25745e9a2940 (table test-table [id=cac58269c8834416bf7d453c009e3245]) successfully deleted
I20260927 05:06:45.585227 8503 raft_consensus.cc:729] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d70f5a7824fd49de2691527de274d, State: Initialized, Role: FOLLOWER
I20260927 05:06:45.582216 8529 ts_tablet_manager.cc:1905] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:45.585672 8503 consensus_queue.cc:260] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
W20260927 05:06:45.585902 7979 catalog_manager.cc:4808] TS c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093): delete failed for tablet 4be86875ab38427a90ce25745e9a2940 because the tablet was not found. No further retry: Not found: Tablet not found: 4be86875ab38427a90ce25745e9a2940
I20260927 05:06:45.586337 8503 ts_tablet_manager.cc:1428] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 05:06:45.588780 8505 tablet_bootstrap.cc:492] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: No bootstrap required, opened a new log
I20260927 05:06:45.589064 8505 ts_tablet_manager.cc:1397] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260927 05:06:45.590067 8505 raft_consensus.cc:348] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.590423 8505 raft_consensus.cc:374] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:45.590590 8505 raft_consensus.cc:729] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83505fa980a94874afb597582cd1106f, State: Initialized, Role: FOLLOWER
I20260927 05:06:45.590956 8505 consensus_queue.cc:260] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.591843 8505 ts_tablet_manager.cc:1428] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260927 05:06:45.600584 8504 tablet_bootstrap.cc:492] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2: No bootstrap required, opened a new log
I20260927 05:06:45.600805 8504 ts_tablet_manager.cc:1397] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.015s
I20260927 05:06:45.601722 8504 raft_consensus.cc:348] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.601970 8504 raft_consensus.cc:374] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:45.602072 8504 raft_consensus.cc:729] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cdeca93e41404597da9c0742b975c2, State: Initialized, Role: FOLLOWER
I20260927 05:06:45.602396 8504 consensus_queue.cc:260] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.604231 8504 ts_tablet_manager.cc:1428] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:45.604763 8529 ts_tablet_manager.cc:1918] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:45.604959 8529 log.cc:1196] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/4be86875ab38427a90ce25745e9a2940
I20260927 05:06:45.605575 8529 ts_tablet_manager.cc:1939] T 4be86875ab38427a90ce25745e9a2940 P 83505fa980a94874afb597582cd1106f: Deleting consensus metadata
I20260927 05:06:45.607546 7979 catalog_manager.cc:4841] TS 83505fa980a94874afb597582cd1106f (127.0.97.2:43505): tablet 4be86875ab38427a90ce25745e9a2940 (table test-table [id=cac58269c8834416bf7d453c009e3245]) successfully deleted
I20260927 05:06:45.769496 8509 raft_consensus.cc:482] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:45.769745 8509 raft_consensus.cc:504] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.771843 8509 leader_election.cc:288] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83505fa980a94874afb597582cd1106f (127.0.97.2:43505), c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093)
I20260927 05:06:45.780014 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.773243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42233 (local address 127.0.97.3:45093)
0927 05:06:45.773435 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.773440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.774079 (+ 639us) server_negotiation.cc:407] Connection header received
0927 05:06:45.774278 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.774284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.774372 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.774456 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:45.774947 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.775658 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.777833 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.778925 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.779053 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.779159 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.779541 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.779684 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:06:45.780061 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.773063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45093 (local address 127.0.97.1:42233)
0927 05:06:45.773936 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:45.773959 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:45.774110 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:45.774467 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:45.774472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:45.774493 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:45.774796 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.774807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.775791 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.775799 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.777656 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.777673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.779148 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.779155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.779340 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.779392 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 05:06:45.779497 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 05:06:45.779663 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":542,"threads_started":1}
I20260927 05:06:45.780766 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb764202b3a5425fa2bdff8ef7e2d057" candidate_uuid: "830d70f5a7824fd49de2691527de274d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2cdeca93e41404597da9c0742b975c2" is_pre_election: true
I20260927 05:06:45.781033 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.772181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52583 (local address 127.0.97.2:43505)
0927 05:06:45.772439 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.772449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.773034 (+ 585us) server_negotiation.cc:407] Connection header received
0927 05:06:45.773361 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.773370 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.773476 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.773577 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:45.774433 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.775658 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.778404 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.779602 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.779741 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.779835 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.780593 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.780732 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:06:45.781028 8434 raft_consensus.cc:2454] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d70f5a7824fd49de2691527de274d in term 0.
I20260927 05:06:45.781265 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.772140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43505 (local address 127.0.97.1:52583)
0927 05:06:45.772909 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:45.772930 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:45.773147 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:45.773800 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:45.773805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:45.773837 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:45.774243 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.774256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.775853 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.775859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.778226 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.778241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.780024 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.780033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.780407 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.780450 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 05:06:45.780541 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 05:06:45.780674 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":496,"threads_started":1}
I20260927 05:06:45.781603 8082 leader_election.cc:302] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d70f5a7824fd49de2691527de274d, c2cdeca93e41404597da9c0742b975c2; no voters:
I20260927 05:06:45.782028 8509 raft_consensus.cc:2781] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:06:45.782150 8509 raft_consensus.cc:482] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:06:45.782244 8509 raft_consensus.cc:3037] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:45.782609 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb764202b3a5425fa2bdff8ef7e2d057" candidate_uuid: "830d70f5a7824fd49de2691527de274d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83505fa980a94874afb597582cd1106f" is_pre_election: true
I20260927 05:06:45.782958 8289 raft_consensus.cc:2454] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d70f5a7824fd49de2691527de274d in term 0.
I20260927 05:06:45.785534 8509 raft_consensus.cc:504] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.786216 8509 leader_election.cc:288] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [CANDIDATE]: Term 1 election: Requested vote from peers 83505fa980a94874afb597582cd1106f (127.0.97.2:43505), c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093)
I20260927 05:06:45.786677 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb764202b3a5425fa2bdff8ef7e2d057" candidate_uuid: "830d70f5a7824fd49de2691527de274d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83505fa980a94874afb597582cd1106f"
I20260927 05:06:45.786895 8289 raft_consensus.cc:3037] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:45.786737 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb764202b3a5425fa2bdff8ef7e2d057" candidate_uuid: "830d70f5a7824fd49de2691527de274d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2cdeca93e41404597da9c0742b975c2"
I20260927 05:06:45.787075 8434 raft_consensus.cc:3037] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Advancing to term 1
W20260927 05:06:45.787919 8340 tablet.cc:2369] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:06:45.789340 8434 raft_consensus.cc:2454] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d70f5a7824fd49de2691527de274d in term 1.
I20260927 05:06:45.789783 8082 leader_election.cc:302] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d70f5a7824fd49de2691527de274d, c2cdeca93e41404597da9c0742b975c2; no voters:
I20260927 05:06:45.790103 8509 raft_consensus.cc:2781] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:45.790241 8289 raft_consensus.cc:2454] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d70f5a7824fd49de2691527de274d in term 1.
I20260927 05:06:45.790642 8509 raft_consensus.cc:686] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 1 LEADER]: Becoming Leader. State: Replica: 830d70f5a7824fd49de2691527de274d, State: Running, Role: LEADER
I20260927 05:06:45.791047 8509 consensus_queue.cc:237] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.794538 7992 catalog_manager.cc:5495] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d reported cstate change: term changed from 0 to 1, leader changed from <none> to 830d70f5a7824fd49de2691527de274d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "830d70f5a7824fd49de2691527de274d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } health_report { overall_health: UNKNOWN } } }
Sep 27 05:06:45 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:45.856650 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.842479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.0.97.62:38825)
0927 05:06:45.843396 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:45.843403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:45.843432 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:06:45.843478 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:45.843480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:45.843540 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:45.843653 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:45.844842 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.845615 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.846857 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.847250 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.847387 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.850878 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:45.850883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:45.850886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:45.850915 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:45.853696 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:45.854566 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:45.854572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:45.854576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:45.854698 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:45.855065 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:45.855072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:45.855075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:45.855349 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:45.855502 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:45.855918 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 05:06:45.856301 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":788,"thread_start_us":631,"threads_started":1}
I20260927 05:06:45.856648 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:45.842387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38825 (local address 127.0.0.1:38570)
0927 05:06:45.843098 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:45.843125 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:45.843297 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:45.843678 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:45.843682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:45.844463 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:45.844696 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.844712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.845758 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.845763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.846714 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:45.846725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:45.847408 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:45.847414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:45.847628 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:45.848356 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:45.848372 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:45.850671 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:45.853846 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:45.853861 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:45.853869 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:45.854432 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:45.854815 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:45.854820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:45.854823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:45.854960 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:45.855492 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:45.855500 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:45.855662 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 05:06:45.855846 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 05:06:45.856179 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":371,"threads_started":1}
I20260927 05:06:45.862809 7992 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38570:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 05:06:45.864077 7992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:06:45.872848 8129 tablet_service.cc:1467] Processing CreateTablet for tablet 6690a496913f431e9bded6542ad44a90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9183221fb1064adb951dccd720895253]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:06:45.872951 8269 tablet_service.cc:1467] Processing CreateTablet for tablet 6690a496913f431e9bded6542ad44a90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9183221fb1064adb951dccd720895253]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:06:45.873543 8129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6690a496913f431e9bded6542ad44a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:45.873557 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6690a496913f431e9bded6542ad44a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:45.881561 8505 tablet_bootstrap.cc:492] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: Bootstrap starting.
I20260927 05:06:45.882215 8409 tablet_service.cc:1467] Processing CreateTablet for tablet 6690a496913f431e9bded6542ad44a90 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9183221fb1064adb951dccd720895253]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 05:06:45.882859 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6690a496913f431e9bded6542ad44a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:45.883667 8505 tablet_bootstrap.cc:654] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:45.883994 8503 tablet_bootstrap.cc:492] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: Bootstrap starting.
I20260927 05:06:45.886027 8503 tablet_bootstrap.cc:654] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:45.893718 8504 tablet_bootstrap.cc:492] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: Bootstrap starting.
I20260927 05:06:45.894932 8503 tablet_bootstrap.cc:492] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: No bootstrap required, opened a new log
I20260927 05:06:45.895135 8503 ts_tablet_manager.cc:1397] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260927 05:06:45.895704 8504 tablet_bootstrap.cc:654] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:45.896121 8503 raft_consensus.cc:348] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.896420 8503 raft_consensus.cc:374] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:45.896529 8503 raft_consensus.cc:729] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d70f5a7824fd49de2691527de274d, State: Initialized, Role: FOLLOWER
I20260927 05:06:45.896868 8503 consensus_queue.cc:260] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.897534 8509 tablet_replica.cc:439] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } }
I20260927 05:06:45.897914 8509 tablet_replica.cc:442] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:06:45.902565 8503 ts_tablet_manager.cc:1428] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20260927 05:06:45.905689 8504 tablet_bootstrap.cc:492] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: No bootstrap required, opened a new log
I20260927 05:06:45.905905 8504 ts_tablet_manager.cc:1397] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260927 05:06:45.906872 8504 raft_consensus.cc:348] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.907156 8504 raft_consensus.cc:374] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:45.907246 8504 raft_consensus.cc:729] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cdeca93e41404597da9c0742b975c2, State: Initialized, Role: FOLLOWER
I20260927 05:06:45.907562 8504 consensus_queue.cc:260] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.908114 8511 tablet_replica.cc:439] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } }
I20260927 05:06:45.908486 8511 tablet_replica.cc:442] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:06:45.909350 8504 ts_tablet_manager.cc:1428] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 05:06:45.911818 8505 tablet_bootstrap.cc:492] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: No bootstrap required, opened a new log
I20260927 05:06:45.912079 8505 ts_tablet_manager.cc:1397] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.014s
I20260927 05:06:45.913029 8505 raft_consensus.cc:348] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.913301 8505 raft_consensus.cc:374] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:45.913414 8505 raft_consensus.cc:729] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83505fa980a94874afb597582cd1106f, State: Initialized, Role: FOLLOWER
I20260927 05:06:45.913695 8505 consensus_queue.cc:260] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:45.914399 8505 ts_tablet_manager.cc:1428] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 05:06:45.914249 8510 tablet_replica.cc:439] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } }
I20260927 05:06:45.914633 8510 tablet_replica.cc:442] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:06:46.177809 8509 consensus_queue.cc:1035] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:46.184525 8549 consensus_queue.cc:1035] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:46.239787 8511 raft_consensus.cc:482] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:46.240119 8511 raft_consensus.cc:504] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.241179 8549 raft_consensus.cc:482] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:46.241415 8549 raft_consensus.cc:504] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.242084 8511 leader_election.cc:288] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653), 83505fa980a94874afb597582cd1106f (127.0.97.2:43505)
I20260927 05:06:46.243026 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6690a496913f431e9bded6542ad44a90" candidate_uuid: "830d70f5a7824fd49de2691527de274d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2cdeca93e41404597da9c0742b975c2" is_pre_election: true
I20260927 05:06:46.243330 8434 raft_consensus.cc:2454] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d70f5a7824fd49de2691527de274d in term 0.
I20260927 05:06:46.243330 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6690a496913f431e9bded6542ad44a90" candidate_uuid: "830d70f5a7824fd49de2691527de274d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83505fa980a94874afb597582cd1106f" is_pre_election: true
I20260927 05:06:46.243654 8289 raft_consensus.cc:2454] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d70f5a7824fd49de2691527de274d in term 0.
I20260927 05:06:46.244023 8082 leader_election.cc:302] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d70f5a7824fd49de2691527de274d, c2cdeca93e41404597da9c0742b975c2; no voters:
I20260927 05:06:46.244478 8535 raft_consensus.cc:2781] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:06:46.244653 8535 raft_consensus.cc:482] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:06:46.244819 8535 raft_consensus.cc:3037] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:46.249521 8535 raft_consensus.cc:504] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.251606 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6690a496913f431e9bded6542ad44a90" candidate_uuid: "830d70f5a7824fd49de2691527de274d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83505fa980a94874afb597582cd1106f"
I20260927 05:06:46.252161 8289 raft_consensus.cc:3037] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:46.252990 8535 leader_election.cc:288] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [CANDIDATE]: Term 1 election: Requested vote from peers 83505fa980a94874afb597582cd1106f (127.0.97.2:43505), c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093)
I20260927 05:06:46.253191 8549 leader_election.cc:288] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83505fa980a94874afb597582cd1106f (127.0.97.2:43505), c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093)
I20260927 05:06:46.253652 8434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6690a496913f431e9bded6542ad44a90" candidate_uuid: "830d70f5a7824fd49de2691527de274d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2cdeca93e41404597da9c0742b975c2"
I20260927 05:06:46.254101 8434 raft_consensus.cc:3037] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:46.256697 8289 raft_consensus.cc:2454] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d70f5a7824fd49de2691527de274d in term 1.
I20260927 05:06:46.257416 8082 leader_election.cc:302] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d70f5a7824fd49de2691527de274d, 83505fa980a94874afb597582cd1106f; no voters:
I20260927 05:06:46.259083 8434 raft_consensus.cc:2454] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d70f5a7824fd49de2691527de274d in term 1.
I20260927 05:06:46.260159 8549 raft_consensus.cc:2781] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:46.260628 8549 raft_consensus.cc:686] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [term 1 LEADER]: Becoming Leader. State: Replica: 830d70f5a7824fd49de2691527de274d, State: Running, Role: LEADER
I20260927 05:06:46.261034 8549 consensus_queue.cc:237] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.261281 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.242324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.97.3:51027)
0927 05:06:46.253119 (+ 10795us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:46.253143 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:46.253410 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:46.253969 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:46.253975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:46.254001 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:46.254329 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:46.254340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.256034 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.256041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:46.258405 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:46.258425 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.260350 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.260356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:46.260582 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.260621 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 05:06:46.260713 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 05:06:46.260863 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9307,"thread_start_us":17999,"threads_started":1}
I20260927 05:06:46.261994 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.242335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51027 (local address 127.0.97.1:36653)
0927 05:06:46.252339 (+ 10004us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:46.252375 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:46.253331 (+ 956us) server_negotiation.cc:407] Connection header received
0927 05:06:46.253575 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:46.253580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:46.253649 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:46.253732 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:46.254621 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.255816 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.258615 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.259985 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.260109 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.260232 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:46.261526 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0927 05:06:46.261685 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9644,"thread_start_us":10620,"threads_started":1}
I20260927 05:06:46.262661 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6690a496913f431e9bded6542ad44a90" candidate_uuid: "c2cdeca93e41404597da9c0742b975c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d70f5a7824fd49de2691527de274d" is_pre_election: true
I20260927 05:06:46.265389 8535 tablet_replica.cc:439] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: TxnStatusTablet state changed. Reason: New leader 830d70f5a7824fd49de2691527de274d. Latest consensus state: current_term: 1 leader_uuid: "830d70f5a7824fd49de2691527de274d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } }
I20260927 05:06:46.265743 8535 tablet_replica.cc:442] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:06:46.266167 7992 catalog_manager.cc:5495] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d reported cstate change: term changed from 0 to 1, leader changed from <none> to 830d70f5a7824fd49de2691527de274d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "830d70f5a7824fd49de2691527de274d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } health_report { overall_health: UNKNOWN } } }
I20260927 05:06:46.268177 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.242789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44349 (local address 127.0.97.2:43505)
0927 05:06:46.243109 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:46.243118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:46.260111 (+ 16993us) server_negotiation.cc:407] Connection header received
0927 05:06:46.260347 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:46.260354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:46.260458 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:46.260552 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:06:46.261795 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.262892 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.265624 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.266887 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.266990 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.267099 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:46.267716 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 05:06:46.267883 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260927 05:06:46.268860 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.242616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43505 (local address 127.0.97.3:44349)
0927 05:06:46.259988 (+ 17372us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:46.260014 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:46.260190 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:46.260904 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:46.260913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:46.260946 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:06:46.261585 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:46.261601 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.263281 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.263286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:46.265409 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:46.265425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.267300 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.267307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:46.267522 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.267559 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:06:46.268369 (+ 810us) client_negotiation.cc:240] Negotiation successful
0927 05:06:46.268530 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17204,"thread_start_us":8840,"threads_started":1}
I20260927 05:06:46.269604 8557 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 05:06:46.269595 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6690a496913f431e9bded6542ad44a90" candidate_uuid: "c2cdeca93e41404597da9c0742b975c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83505fa980a94874afb597582cd1106f" is_pre_election: true
I20260927 05:06:46.269861 8557 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 05:06:46.269941 8289 raft_consensus.cc:2379] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2cdeca93e41404597da9c0742b975c2 in current term 1: Already voted for candidate 830d70f5a7824fd49de2691527de274d in this term.
I20260927 05:06:46.270638 8362 leader_election.cc:302] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2cdeca93e41404597da9c0742b975c2; no voters: 830d70f5a7824fd49de2691527de274d, 83505fa980a94874afb597582cd1106f
I20260927 05:06:46.271471 8511 raft_consensus.cc:2726] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 05:06:46.350574 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.344054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.0.97.1:36653)
0927 05:06:46.344286 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:46.344293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:46.344441 (+ 148us) server_negotiation.cc:407] Connection header received
0927 05:06:46.344741 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:46.344745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:46.344830 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:46.344957 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:06:46.345347 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.346121 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.348138 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.348622 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.348720 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.349327 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:06:46.349956 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:06:46.350088 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:46.350241 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 05:06:46.350320 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:06:46.350769 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.344051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.0.1:53012)
0927 05:06:46.344321 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:46.344344 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:46.344535 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:46.345001 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:46.345006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:46.345020 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:06:46.345215 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:46.345225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.346272 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.346278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:46.347920 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:46.347979 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.348781 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.348787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:46.349019 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.349105 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:06:46.350084 (+ 979us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:06:46.350108 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 05:06:46.350208 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 05:06:46.350402 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 05:06:46.355805 8434 raft_consensus.cc:1261] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Refusing update from remote peer 830d70f5a7824fd49de2691527de274d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:46.355856 8289 raft_consensus.cc:1261] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Refusing update from remote peer 830d70f5a7824fd49de2691527de274d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:46.356572 8549 consensus_queue.cc:1035] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [LEADER]: Connected to new peer: Peer: permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:46.356854 8535 consensus_queue.cc:1035] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:46.361011 8510 tablet_replica.cc:439] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: TxnStatusTablet state changed. Reason: New leader 830d70f5a7824fd49de2691527de274d. Latest consensus state: current_term: 1 leader_uuid: "830d70f5a7824fd49de2691527de274d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } }
I20260927 05:06:46.361331 8510 tablet_replica.cc:442] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:06:46.363268 8552 mvcc.cc:205] Tried to move back new op lower bound from 7333829043625738240 to 7333829043250507776. Current Snapshot: MvccSnapshot[applied={T|T < 7333829043625738240}]
I20260927 05:06:46.367543 8510 tablet_replica.cc:439] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "830d70f5a7824fd49de2691527de274d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } }
I20260927 05:06:46.367820 8510 tablet_replica.cc:442] T 6690a496913f431e9bded6542ad44a90 P 83505fa980a94874afb597582cd1106f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:06:46.372766 8535 tablet_replica.cc:439] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "830d70f5a7824fd49de2691527de274d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } }
I20260927 05:06:46.373165 8535 tablet_replica.cc:442] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:06:46.375689 8511 tablet_replica.cc:439] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: TxnStatusTablet state changed. Reason: New leader 830d70f5a7824fd49de2691527de274d. Latest consensus state: current_term: 1 leader_uuid: "830d70f5a7824fd49de2691527de274d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } }
I20260927 05:06:46.376000 8511 tablet_replica.cc:442] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:06:46.379647 8549 tablet_replica.cc:439] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "830d70f5a7824fd49de2691527de274d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } }
I20260927 05:06:46.380121 8549 tablet_replica.cc:442] T 6690a496913f431e9bded6542ad44a90 P 830d70f5a7824fd49de2691527de274d: This TxnStatusTablet replica's current role is: LEADER
I20260927 05:06:46.382329 8511 tablet_replica.cc:439] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "830d70f5a7824fd49de2691527de274d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } } }
I20260927 05:06:46.382643 8511 tablet_replica.cc:442] T 6690a496913f431e9bded6542ad44a90 P c2cdeca93e41404597da9c0742b975c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 05:06:46.394878 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.389262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.0.97.1:36653)
0927 05:06:46.389526 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:46.389533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:46.389817 (+ 284us) server_negotiation.cc:407] Connection header received
0927 05:06:46.390046 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:46.390060 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:46.390133 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:46.390245 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:06:46.390726 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.391579 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.393105 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.393572 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.393710 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.394109 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:06:46.394223 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:06:46.394329 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:46.394476 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 05:06:46.394559 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 05:06:46.413628 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.407324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59375 (local address 127.0.97.1:36653)
0927 05:06:46.407498 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:46.407503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:46.408067 (+ 564us) server_negotiation.cc:407] Connection header received
0927 05:06:46.408301 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:46.408306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:46.408374 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:46.408470 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:06:46.408943 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.409815 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.411527 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.412049 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.412208 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.412697 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:06:46.412845 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:06:46.412947 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:46.413108 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 05:06:46.413227 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":49}
I20260927 05:06:46.413733 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.407375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.97.1:59375)
0927 05:06:46.407900 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:46.407919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:46.408162 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:46.408479 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:46.408484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:46.408494 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:06:46.408774 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:46.408783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.409975 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.409981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:46.411340 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:46.411356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.412224 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.412230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:46.412399 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.412485 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:06:46.412946 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:06:46.412957 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 05:06:46.413090 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 05:06:46.413276 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":45,"thread_start_us":261,"threads_started":1}
I20260927 05:06:46.500442 7990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:46.500715 7990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:46.504331 7990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1: Sending DeleteTablet for 3 replicas of tablet cb764202b3a5425fa2bdff8ef7e2d057
I20260927 05:06:46.505369 8128 tablet_service.cc:1514] Processing DeleteTablet for tablet cb764202b3a5425fa2bdff8ef7e2d057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53002
I20260927 05:06:46.505546 8269 tablet_service.cc:1514] Processing DeleteTablet for tablet cb764202b3a5425fa2bdff8ef7e2d057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43218
I20260927 05:06:46.506001 8409 tablet_service.cc:1514] Processing DeleteTablet for tablet cb764202b3a5425fa2bdff8ef7e2d057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41474
I20260927 05:06:46.523186 8573 tablet_replica.cc:330] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: stopping tablet replica
I20260927 05:06:46.523516 8573 raft_consensus.cc:2229] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:46.523821 8573 raft_consensus.cc:2258] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:46.525425 8573 ts_tablet_manager.cc:1905] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:46.528230 8574 tablet_replica.cc:330] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2: stopping tablet replica
I20260927 05:06:46.528674 8574 raft_consensus.cc:2229] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:46.529093 8574 raft_consensus.cc:2258] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:46.529682 8572 tablet_replica.cc:330] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d: stopping tablet replica
I20260927 05:06:46.531147 8574 ts_tablet_manager.cc:1905] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:46.531520 8572 raft_consensus.cc:2229] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:06:46.532349 8572 raft_consensus.cc:2258] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:46.534756 8573 ts_tablet_manager.cc:1918] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:06:46.534988 8573 log.cc:1196] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/cb764202b3a5425fa2bdff8ef7e2d057
I20260927 05:06:46.535671 8573 ts_tablet_manager.cc:1939] T cb764202b3a5425fa2bdff8ef7e2d057 P 83505fa980a94874afb597582cd1106f: Deleting consensus metadata
I20260927 05:06:46.536756 8572 ts_tablet_manager.cc:1905] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:46.537283 7979 catalog_manager.cc:4841] TS 83505fa980a94874afb597582cd1106f (127.0.97.2:43505): tablet cb764202b3a5425fa2bdff8ef7e2d057 (table test-table [id=6a9cb8d084ed47efb86c57fb516b8976]) successfully deleted
I20260927 05:06:46.545508 8572 ts_tablet_manager.cc:1918] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:06:46.545863 8572 log.cc:1196] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/cb764202b3a5425fa2bdff8ef7e2d057
I20260927 05:06:46.546389 8574 ts_tablet_manager.cc:1918] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:06:46.546589 8574 log.cc:1196] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/cb764202b3a5425fa2bdff8ef7e2d057
I20260927 05:06:46.546840 8572 ts_tablet_manager.cc:1939] T cb764202b3a5425fa2bdff8ef7e2d057 P 830d70f5a7824fd49de2691527de274d: Deleting consensus metadata
I20260927 05:06:46.547246 8574 ts_tablet_manager.cc:1939] T cb764202b3a5425fa2bdff8ef7e2d057 P c2cdeca93e41404597da9c0742b975c2: Deleting consensus metadata
I20260927 05:06:46.548493 7977 catalog_manager.cc:4841] TS 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653): tablet cb764202b3a5425fa2bdff8ef7e2d057 (table test-table [id=6a9cb8d084ed47efb86c57fb516b8976]) successfully deleted
I20260927 05:06:46.548460 7979 catalog_manager.cc:4841] TS c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093): tablet cb764202b3a5425fa2bdff8ef7e2d057 (table test-table [id=6a9cb8d084ed47efb86c57fb516b8976]) successfully deleted
Sep 27 05:06:46 dist-test-slave-hgs0 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 05:06:46 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:46 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:06:46.594208 8582 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:46.580290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.0.97.62:38825)
0927 05:06:46.581231 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:46.581237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:46.581253 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:06:46.581306 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:46.581310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:46.581367 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:46.581522 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:46.582887 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.583693 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.585024 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.585518 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.585665 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.593867 (+ 8202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38584: BlockingRecv error: recv got EOF from 127.0.0.1:38584 (error 108)
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":646,"threads_started":1}
Sep 27 05:06:46 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260927 05:06:46.613003 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.597033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.0.97.62:38825)
0927 05:06:46.597235 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:46.597239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:46.597634 (+ 395us) server_negotiation.cc:407] Connection header received
0927 05:06:46.597864 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:46.597869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:46.597932 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:46.598043 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:46.599213 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.599929 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.601350 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.601764 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.601894 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.606184 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:46.606191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:46.606196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:46.606242 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:46.609853 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:46.610644 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:46.610650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:46.610654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:46.610827 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:46.611311 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:46.611316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:46.611320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:46.611715 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:46.612025 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:46.612384 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 05:06:46.612643 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 05:06:46.617046 7990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:06:46.617839 7990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:06:46.625820 8128 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6770102e2346ec9c55dc5cab66d724 (DEFAULT_TABLE table=test-table [id=e25510af87064bf29ef6bcb3498a7f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:46.626227 8269 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6770102e2346ec9c55dc5cab66d724 (DEFAULT_TABLE table=test-table [id=e25510af87064bf29ef6bcb3498a7f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:46.626466 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6770102e2346ec9c55dc5cab66d724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:46.626698 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6770102e2346ec9c55dc5cab66d724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:46.633092 8588 tablet_bootstrap.cc:492] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f: Bootstrap starting.
I20260927 05:06:46.633333 8409 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6770102e2346ec9c55dc5cab66d724 (DEFAULT_TABLE table=test-table [id=e25510af87064bf29ef6bcb3498a7f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:46.633823 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6770102e2346ec9c55dc5cab66d724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:46.636543 8588 tablet_bootstrap.cc:654] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:46.644773 8591 tablet_bootstrap.cc:492] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2: Bootstrap starting.
I20260927 05:06:46.646034 8588 tablet_bootstrap.cc:492] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f: No bootstrap required, opened a new log
I20260927 05:06:46.646317 8588 ts_tablet_manager.cc:1397] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260927 05:06:46.646705 8591 tablet_bootstrap.cc:654] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:46.647315 8588 raft_consensus.cc:348] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.647670 8588 raft_consensus.cc:374] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:46.647825 8588 raft_consensus.cc:729] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83505fa980a94874afb597582cd1106f, State: Initialized, Role: FOLLOWER
I20260927 05:06:46.648221 8588 consensus_queue.cc:260] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.648913 8589 tablet_bootstrap.cc:492] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d: Bootstrap starting.
I20260927 05:06:46.649663 8588 ts_tablet_manager.cc:1428] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 05:06:46.650195 8591 tablet_bootstrap.cc:492] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2: No bootstrap required, opened a new log
I20260927 05:06:46.650333 8591 ts_tablet_manager.cc:1397] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 05:06:46.651104 8591 raft_consensus.cc:348] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.651325 8591 raft_consensus.cc:374] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:46.651422 8591 raft_consensus.cc:729] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cdeca93e41404597da9c0742b975c2, State: Initialized, Role: FOLLOWER
I20260927 05:06:46.651635 8591 consensus_queue.cc:260] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.653586 8591 ts_tablet_manager.cc:1428] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 05:06:46.653939 8589 tablet_bootstrap.cc:654] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:46.656482 8589 tablet_bootstrap.cc:492] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d: No bootstrap required, opened a new log
I20260927 05:06:46.656661 8589 ts_tablet_manager.cc:1397] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 05:06:46.657310 8589 raft_consensus.cc:348] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.657580 8589 raft_consensus.cc:374] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:46.657686 8589 raft_consensus.cc:729] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d70f5a7824fd49de2691527de274d, State: Initialized, Role: FOLLOWER
I20260927 05:06:46.657949 8589 consensus_queue.cc:260] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.658529 8589 ts_tablet_manager.cc:1428] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 05:06:46.758342 8511 raft_consensus.cc:482] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:46.758592 8511 raft_consensus.cc:504] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.759312 8511 leader_election.cc:288] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653), 83505fa980a94874afb597582cd1106f (127.0.97.2:43505)
I20260927 05:06:46.759907 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6770102e2346ec9c55dc5cab66d724" candidate_uuid: "c2cdeca93e41404597da9c0742b975c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83505fa980a94874afb597582cd1106f" is_pre_election: true
I20260927 05:06:46.759958 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6770102e2346ec9c55dc5cab66d724" candidate_uuid: "c2cdeca93e41404597da9c0742b975c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d70f5a7824fd49de2691527de274d" is_pre_election: true
I20260927 05:06:46.760270 8289 raft_consensus.cc:2454] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2cdeca93e41404597da9c0742b975c2 in term 0.
I20260927 05:06:46.760305 8149 raft_consensus.cc:2454] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2cdeca93e41404597da9c0742b975c2 in term 0.
I20260927 05:06:46.760859 8362 leader_election.cc:302] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 830d70f5a7824fd49de2691527de274d, 83505fa980a94874afb597582cd1106f, c2cdeca93e41404597da9c0742b975c2; no voters:
I20260927 05:06:46.761193 8511 raft_consensus.cc:2781] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:06:46.761335 8511 raft_consensus.cc:482] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:06:46.761461 8511 raft_consensus.cc:3037] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:46.763748 8511 raft_consensus.cc:504] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.764494 8511 leader_election.cc:288] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [CANDIDATE]: Term 1 election: Requested vote from peers 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653), 83505fa980a94874afb597582cd1106f (127.0.97.2:43505)
I20260927 05:06:46.764989 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6770102e2346ec9c55dc5cab66d724" candidate_uuid: "c2cdeca93e41404597da9c0742b975c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d70f5a7824fd49de2691527de274d"
I20260927 05:06:46.765033 8289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6770102e2346ec9c55dc5cab66d724" candidate_uuid: "c2cdeca93e41404597da9c0742b975c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83505fa980a94874afb597582cd1106f"
I20260927 05:06:46.765228 8149 raft_consensus.cc:3037] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:46.765288 8289 raft_consensus.cc:3037] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:46.767688 8149 raft_consensus.cc:2454] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cdeca93e41404597da9c0742b975c2 in term 1.
I20260927 05:06:46.768236 8360 leader_election.cc:302] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d70f5a7824fd49de2691527de274d, c2cdeca93e41404597da9c0742b975c2; no voters:
I20260927 05:06:46.768589 8511 raft_consensus.cc:2781] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:46.768595 8289 raft_consensus.cc:2454] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cdeca93e41404597da9c0742b975c2 in term 1.
I20260927 05:06:46.769522 8511 raft_consensus.cc:686] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 1 LEADER]: Becoming Leader. State: Replica: c2cdeca93e41404597da9c0742b975c2, State: Running, Role: LEADER
I20260927 05:06:46.769913 8511 consensus_queue.cc:237] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.773659 7990 catalog_manager.cc:5495] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2cdeca93e41404597da9c0742b975c2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c2cdeca93e41404597da9c0742b975c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } health_report { overall_health: HEALTHY } } }
I20260927 05:06:46.795608 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.788557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.0.97.3:45093)
0927 05:06:46.789186 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:46.789192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:46.789220 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:06:46.789339 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:46.789347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:46.789471 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:46.789681 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:06:46.790124 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.791298 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.792819 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:46.793494 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:46.793632 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:46.794034 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:06:46.794892 (+ 858us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:06:46.795069 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:46.795148 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 05:06:46.795258 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":317,"threads_started":1}
I20260927 05:06:46.799753 8149 raft_consensus.cc:1261] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Refusing update from remote peer c2cdeca93e41404597da9c0742b975c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:46.799750 8289 raft_consensus.cc:1261] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Refusing update from remote peer c2cdeca93e41404597da9c0742b975c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:46.800506 8594 consensus_queue.cc:1035] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:46.800840 8511 consensus_queue.cc:1035] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:46.836961 7992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:46.837271 7992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:46.841112 7992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1: Sending DeleteTablet for 3 replicas of tablet 3a6770102e2346ec9c55dc5cab66d724
I20260927 05:06:46.842329 8409 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a6770102e2346ec9c55dc5cab66d724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41474
I20260927 05:06:46.842410 8128 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a6770102e2346ec9c55dc5cab66d724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53002
I20260927 05:06:46.842629 8574 tablet_replica.cc:330] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2: stopping tablet replica
I20260927 05:06:46.842754 8269 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a6770102e2346ec9c55dc5cab66d724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43218
I20260927 05:06:46.842835 8574 raft_consensus.cc:2229] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:06:46.843042 8572 tablet_replica.cc:330] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d: stopping tablet replica
I20260927 05:06:46.843241 8572 raft_consensus.cc:2229] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:46.843362 8574 raft_consensus.cc:2258] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:46.843459 8572 raft_consensus.cc:2258] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:46.843724 8573 tablet_replica.cc:330] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f: stopping tablet replica
I20260927 05:06:46.843910 8573 raft_consensus.cc:2229] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:46.844748 8573 raft_consensus.cc:2258] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:46.845887 8573 ts_tablet_manager.cc:1905] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:46.848651 8574 ts_tablet_manager.cc:1905] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:46.852872 8572 ts_tablet_manager.cc:1905] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:46.852784 7992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:06:46.853808 7992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:06:46.855753 8574 ts_tablet_manager.cc:1918] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:46.855916 8574 log.cc:1196] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/3a6770102e2346ec9c55dc5cab66d724
I20260927 05:06:46.856469 8574 ts_tablet_manager.cc:1939] T 3a6770102e2346ec9c55dc5cab66d724 P c2cdeca93e41404597da9c0742b975c2: Deleting consensus metadata
I20260927 05:06:46.857724 7979 catalog_manager.cc:4841] TS c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093): tablet 3a6770102e2346ec9c55dc5cab66d724 (table test-table [id=e25510af87064bf29ef6bcb3498a7f70]) successfully deleted
I20260927 05:06:46.862157 8573 ts_tablet_manager.cc:1918] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:46.862382 8573 log.cc:1196] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/3a6770102e2346ec9c55dc5cab66d724
I20260927 05:06:46.862926 8573 ts_tablet_manager.cc:1939] T 3a6770102e2346ec9c55dc5cab66d724 P 83505fa980a94874afb597582cd1106f: Deleting consensus metadata
I20260927 05:06:46.863665 8128 tablet_service.cc:1467] Processing CreateTablet for tablet 54c8bd17824042b39c09e65a856d0cab (DEFAULT_TABLE table=test-table [id=11abcf222cd9471cbd8d2e5167ffb2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:46.864393 7979 catalog_manager.cc:4841] TS 83505fa980a94874afb597582cd1106f (127.0.97.2:43505): tablet 3a6770102e2346ec9c55dc5cab66d724 (table test-table [id=e25510af87064bf29ef6bcb3498a7f70]) successfully deleted
I20260927 05:06:46.865265 8409 tablet_service.cc:1467] Processing CreateTablet for tablet 54c8bd17824042b39c09e65a856d0cab (DEFAULT_TABLE table=test-table [id=11abcf222cd9471cbd8d2e5167ffb2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:46.865850 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c8bd17824042b39c09e65a856d0cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:46.868057 8572 ts_tablet_manager.cc:1918] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:06:46.868278 8572 log.cc:1196] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/3a6770102e2346ec9c55dc5cab66d724
I20260927 05:06:46.868389 8269 tablet_service.cc:1467] Processing CreateTablet for tablet 54c8bd17824042b39c09e65a856d0cab (DEFAULT_TABLE table=test-table [id=11abcf222cd9471cbd8d2e5167ffb2ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:06:46.868924 8269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c8bd17824042b39c09e65a856d0cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:46.868886 8572 ts_tablet_manager.cc:1939] T 3a6770102e2346ec9c55dc5cab66d724 P 830d70f5a7824fd49de2691527de274d: Deleting consensus metadata
I20260927 05:06:46.871345 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c8bd17824042b39c09e65a856d0cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:06:46.875779 8588 tablet_bootstrap.cc:492] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f: Bootstrap starting.
I20260927 05:06:46.877660 8588 tablet_bootstrap.cc:654] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:46.885552 8591 tablet_bootstrap.cc:492] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2: Bootstrap starting.
I20260927 05:06:46.886140 8589 tablet_bootstrap.cc:492] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d: Bootstrap starting.
I20260927 05:06:46.887634 8591 tablet_bootstrap.cc:654] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:46.888295 8589 tablet_bootstrap.cc:654] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:46.888748 8588 tablet_bootstrap.cc:492] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f: No bootstrap required, opened a new log
I20260927 05:06:46.888957 8588 ts_tablet_manager.cc:1397] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20260927 05:06:46.889832 8588 raft_consensus.cc:348] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.890132 8588 raft_consensus.cc:374] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:46.890220 8588 raft_consensus.cc:729] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83505fa980a94874afb597582cd1106f, State: Initialized, Role: FOLLOWER
I20260927 05:06:46.890507 8588 consensus_queue.cc:260] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.891124 8588 ts_tablet_manager.cc:1428] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 05:06:46.895890 7977 catalog_manager.cc:4841] TS 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653): tablet 3a6770102e2346ec9c55dc5cab66d724 (table test-table [id=e25510af87064bf29ef6bcb3498a7f70]) successfully deleted
I20260927 05:06:46.896701 8589 tablet_bootstrap.cc:492] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d: No bootstrap required, opened a new log
I20260927 05:06:46.896936 8589 ts_tablet_manager.cc:1397] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260927 05:06:46.897855 8589 raft_consensus.cc:348] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.898136 8589 raft_consensus.cc:374] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:46.898226 8589 raft_consensus.cc:729] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d70f5a7824fd49de2691527de274d, State: Initialized, Role: FOLLOWER
I20260927 05:06:46.898520 8589 consensus_queue.cc:260] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.900288 8589 ts_tablet_manager.cc:1428] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 05:06:46.901990 8591 tablet_bootstrap.cc:492] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2: No bootstrap required, opened a new log
I20260927 05:06:46.902179 8591 ts_tablet_manager.cc:1397] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.011s
I20260927 05:06:46.903035 8591 raft_consensus.cc:348] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.904003 8591 raft_consensus.cc:374] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:46.904102 8591 raft_consensus.cc:729] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cdeca93e41404597da9c0742b975c2, State: Initialized, Role: FOLLOWER
I20260927 05:06:46.904486 8591 consensus_queue.cc:260] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.905097 8591 ts_tablet_manager.cc:1428] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 05:06:46.909521 8601 raft_consensus.cc:482] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:06:46.909760 8601 raft_consensus.cc:504] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.910620 8601 leader_election.cc:288] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653), 83505fa980a94874afb597582cd1106f (127.0.97.2:43505)
I20260927 05:06:46.911043 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c8bd17824042b39c09e65a856d0cab" candidate_uuid: "c2cdeca93e41404597da9c0742b975c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d70f5a7824fd49de2691527de274d" is_pre_election: true
I20260927 05:06:46.911342 8149 raft_consensus.cc:2454] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2cdeca93e41404597da9c0742b975c2 in term 0.
I20260927 05:06:46.911306 8288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c8bd17824042b39c09e65a856d0cab" candidate_uuid: "c2cdeca93e41404597da9c0742b975c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83505fa980a94874afb597582cd1106f" is_pre_election: true
I20260927 05:06:46.911574 8288 raft_consensus.cc:2454] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2cdeca93e41404597da9c0742b975c2 in term 0.
I20260927 05:06:46.911852 8360 leader_election.cc:302] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d70f5a7824fd49de2691527de274d, c2cdeca93e41404597da9c0742b975c2; no voters:
I20260927 05:06:46.912374 8601 raft_consensus.cc:2781] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:06:46.912521 8601 raft_consensus.cc:482] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:06:46.912624 8601 raft_consensus.cc:3037] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:46.916538 8601 raft_consensus.cc:504] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.917804 8149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c8bd17824042b39c09e65a856d0cab" candidate_uuid: "c2cdeca93e41404597da9c0742b975c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d70f5a7824fd49de2691527de274d"
I20260927 05:06:46.918083 8149 raft_consensus.cc:3037] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:46.921336 8149 raft_consensus.cc:2454] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cdeca93e41404597da9c0742b975c2 in term 1.
I20260927 05:06:46.921907 8360 leader_election.cc:302] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d70f5a7824fd49de2691527de274d, c2cdeca93e41404597da9c0742b975c2; no voters:
I20260927 05:06:46.922230 8594 raft_consensus.cc:2781] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:46.922403 8594 raft_consensus.cc:686] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 1 LEADER]: Becoming Leader. State: Replica: c2cdeca93e41404597da9c0742b975c2, State: Running, Role: LEADER
I20260927 05:06:46.922669 8601 leader_election.cc:288] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [CANDIDATE]: Term 1 election: Requested vote from peers 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653), 83505fa980a94874afb597582cd1106f (127.0.97.2:43505)
I20260927 05:06:46.922744 8594 consensus_queue.cc:237] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } }
I20260927 05:06:46.924052 8288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c8bd17824042b39c09e65a856d0cab" candidate_uuid: "c2cdeca93e41404597da9c0742b975c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83505fa980a94874afb597582cd1106f"
I20260927 05:06:46.924306 8288 raft_consensus.cc:3037] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:46.927455 8288 raft_consensus.cc:2454] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cdeca93e41404597da9c0742b975c2 in term 1.
I20260927 05:06:46.927354 7992 catalog_manager.cc:5495] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2cdeca93e41404597da9c0742b975c2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "c2cdeca93e41404597da9c0742b975c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2cdeca93e41404597da9c0742b975c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45093 } health_report { overall_health: HEALTHY } } }
I20260927 05:06:47.014355 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.979233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36653 (local address 127.0.97.3:47257)
0927 05:06:47.005104 (+ 25871us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:47.005127 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:47.005345 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:47.006145 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:47.006151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:47.006208 (+ 57us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:06:47.006581 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:47.006596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.008668 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.008677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:47.010802 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:47.010821 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.012519 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.012526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:47.012752 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:47.012846 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:06:47.013573 (+ 727us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:06:47.013587 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 05:06:47.013670 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 05:06:47.013851 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25626,"thread_start_us":32752,"threads_started":1}
I20260927 05:06:47.015311 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:46.979767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47257 (local address 127.0.97.1:36653)
0927 05:06:47.004363 (+ 24596us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:47.004382 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:47.005505 (+ 1123us) server_negotiation.cc:407] Connection header received
0927 05:06:47.005595 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:47.005600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:47.005703 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:47.005808 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:06:47.006821 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.008447 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.011025 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.011753 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.011902 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:47.013199 (+ 1297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:06:47.013408 (+ 209us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:06:47.013511 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:47.014803 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0927 05:06:47.014906 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24428,"thread_start_us":24650,"threads_started":1}
I20260927 05:06:47.032133 8149 raft_consensus.cc:1261] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Refusing update from remote peer c2cdeca93e41404597da9c0742b975c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:47.033452 8511 consensus_queue.cc:1035] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d70f5a7824fd49de2691527de274d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:47.037014 8288 raft_consensus.cc:1261] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Refusing update from remote peer c2cdeca93e41404597da9c0742b975c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:06:47.038007 8511 consensus_queue.cc:1035] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83505fa980a94874afb597582cd1106f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:06:47.048414 8550 mvcc.cc:205] Tried to move back new op lower bound from 7333829046391996416 to 7333829045958492160. Current Snapshot: MvccSnapshot[applied={T|T < 7333829046391996416}]
I20260927 05:06:47.098815 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:47.089409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40657 (local address 127.0.97.3:45093)
0927 05:06:47.089584 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:47.089590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:47.092479 (+ 2889us) server_negotiation.cc:407] Connection header received
0927 05:06:47.092645 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:47.092649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:47.092716 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:47.092822 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:06:47.093435 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.094476 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.096693 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.097383 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.097461 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:47.098039 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:06:47.098150 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:06:47.098258 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:47.098402 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 05:06:47.098481 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:06:47.098905 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:47.089389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45093 (local address 127.0.97.1:40657)
0927 05:06:47.092358 (+ 2969us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:47.092381 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:47.092518 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:47.092857 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:47.092861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:47.092885 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 05:06:47.093212 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:47.093227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.094620 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.094623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:47.096527 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:47.096545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.097661 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.097668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:47.097800 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:47.097842 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 05:06:47.098263 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 05:06:47.098277 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 05:06:47.098360 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 05:06:47.098513 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2813,"thread_start_us":2679,"threads_started":1}
I20260927 05:06:47.150516 7992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:47.150822 7992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:06:47.155069 7992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3c208689fd740ce859c98ec4fa324f1: Sending DeleteTablet for 3 replicas of tablet 54c8bd17824042b39c09e65a856d0cab
I20260927 05:06:47.156549 8269 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c8bd17824042b39c09e65a856d0cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43218
I20260927 05:06:47.156936 8573 tablet_replica.cc:330] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f: stopping tablet replica
I20260927 05:06:47.157014 8128 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c8bd17824042b39c09e65a856d0cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53002
I20260927 05:06:47.157153 8573 raft_consensus.cc:2229] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:47.157346 8572 tablet_replica.cc:330] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d: stopping tablet replica
I20260927 05:06:47.157395 8573 raft_consensus.cc:2258] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:47.157564 8572 raft_consensus.cc:2229] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:06:47.157656 8409 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c8bd17824042b39c09e65a856d0cab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:06:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41474
I20260927 05:06:47.157824 8572 raft_consensus.cc:2258] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:47.158159 8574 tablet_replica.cc:330] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2: stopping tablet replica
I20260927 05:06:47.158360 8574 raft_consensus.cc:2229] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:06:47.158756 8574 raft_consensus.cc:2258] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:06:47.159996 8574 ts_tablet_manager.cc:1905] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:47.159988 8573 ts_tablet_manager.cc:1905] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:47.165207 8572 ts_tablet_manager.cc:1905] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:06:47.167296 8574 ts_tablet_manager.cc:1918] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:06:47.167965 8574 log.cc:1196] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/54c8bd17824042b39c09e65a856d0cab
I20260927 05:06:47.168473 8574 ts_tablet_manager.cc:1939] T 54c8bd17824042b39c09e65a856d0cab P c2cdeca93e41404597da9c0742b975c2: Deleting consensus metadata
I20260927 05:06:47.169991 7979 catalog_manager.cc:4841] TS c2cdeca93e41404597da9c0742b975c2 (127.0.97.3:45093): tablet 54c8bd17824042b39c09e65a856d0cab (table test-table [id=11abcf222cd9471cbd8d2e5167ffb2ae]) successfully deleted
I20260927 05:06:47.172415 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 8063
I20260927 05:06:47.192274 8573 ts_tablet_manager.cc:1918] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:06:47.192567 8573 log.cc:1196] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/54c8bd17824042b39c09e65a856d0cab
I20260927 05:06:47.193238 8573 ts_tablet_manager.cc:1939] T 54c8bd17824042b39c09e65a856d0cab P 83505fa980a94874afb597582cd1106f: Deleting consensus metadata
I20260927 05:06:47.195014 7979 catalog_manager.cc:4841] TS 83505fa980a94874afb597582cd1106f (127.0.97.2:43505): tablet 54c8bd17824042b39c09e65a856d0cab (table test-table [id=11abcf222cd9471cbd8d2e5167ffb2ae]) successfully deleted
I20260927 05:06:47.195775 8572 ts_tablet_manager.cc:1918] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 05:06:47.196007 8572 log.cc:1196] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/54c8bd17824042b39c09e65a856d0cab
I20260927 05:06:47.196743 8572 ts_tablet_manager.cc:1939] T 54c8bd17824042b39c09e65a856d0cab P 830d70f5a7824fd49de2691527de274d: Deleting consensus metadata
I20260927 05:06:47.198374 7977 catalog_manager.cc:4841] TS 830d70f5a7824fd49de2691527de274d (127.0.97.1:36653): tablet 54c8bd17824042b39c09e65a856d0cab (table test-table [id=11abcf222cd9471cbd8d2e5167ffb2ae]) successfully deleted
Sep 27 05:06:47 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260927 05:06:47.218927 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:47.174011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.0.97.1:36653)
0927 05:06:47.174214 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:47.174221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:47.199423 (+ 25202us) server_negotiation.cc:407] Connection header received
0927 05:06:47.199650 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:47.199655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:47.199738 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:47.199824 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:47.201554 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.202696 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.204690 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.205430 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.205541 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:47.211146 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:47.211167 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:47.211190 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:47.211240 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:47.215401 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:47.216389 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:47.216399 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:47.216403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:47.216601 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:47.217091 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:47.217096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:47.217100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:47.217588 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:47.217938 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:47.218251 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 05:06:47.218521 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:06:47.219868 8193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:47.404800 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 8063
I20260927 05:06:47.442677 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 8203
Sep 27 05:06:47 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260927 05:06:47.461656 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:47.443575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.0.97.2:43505)
0927 05:06:47.444677 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:47.444681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:47.444699 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:06:47.444757 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:47.444761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:47.444831 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:47.444970 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:47.446303 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.447095 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.448618 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.449096 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.449250 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:47.454738 (+ 5488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:47.454764 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:47.454781 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:47.454825 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:47.458150 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:47.459299 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:47.459308 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:47.459311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:47.459483 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:47.460017 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:47.460021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:47.460024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:47.460437 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:47.460667 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:47.461067 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 05:06:47.461256 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":843,"threads_started":1}
I20260927 05:06:47.462613 8334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:47.591008 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 8203
I20260927 05:06:47.614418 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 8343
Sep 27 05:06:47 dist-test-slave-hgs0 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260927 05:06:47.632505 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:47.615385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.0.97.3:45093)
0927 05:06:47.615921 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:47.615926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:47.616002 (+ 76us) server_negotiation.cc:407] Connection header received
0927 05:06:47.616073 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:47.616077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:47.616151 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:47.616301 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:47.617850 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.618634 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.620173 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:47.620612 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:47.620724 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:47.625458 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:47.625475 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:47.625494 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:47.625537 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:47.628766 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:47.629893 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:47.629907 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:47.629912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:47.630125 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:47.630661 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:47.630665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:47.630667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:47.631099 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:47.631359 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:47.631807 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 05:06:47.632087 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":298,"threads_started":1}
I20260927 05:06:47.633190 8474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:47.760828 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 8343
I20260927 05:06:47.783756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 7960
I20260927 05:06:47.785040 8032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:47.908567 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 7960
2026-09-27T05:06:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6531 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:06:48 dist-test-slave-hgs0 krb5kdc[8626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:06:48 dist-test-slave-hgs0 krb5kdc[8626](info): set up 2 sockets
Sep 27 05:06:48 dist-test-slave-hgs0 krb5kdc[8626](info): commencing operation
krb5kdc: starting...
W20260927 05:06:49.077107 388 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.010s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:06:49 dist-test-slave-hgs0 krb5kdc[8626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:06:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:06:49.328508 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38761
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:49.682101 8639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:49.682487 8639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:49.682660 8639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:49.690291 8639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:49.690460 8639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:49.690570 8639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:49.690671 8639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:49.690752 8639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:49.702323 8639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:49.703907 8639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:49.705541 8639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:49.718498 8644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:49.718568 8645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:49.718570 8647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:49.719321 8639 server_base.cc:1034] running on GCE node
I20260927 05:06:49.720304 8639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:49.722023 8639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:49.723278 8639 hybrid_clock.cc:648] HybridClock initialized: now 1790485609723215 us; error 60 us; skew 500 ppm
I20260927 05:06:49.723692 8639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T05:06:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1887 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:06:49 dist-test-slave-hgs0 krb5kdc[8653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:06:49 dist-test-slave-hgs0 krb5kdc[8653](info): set up 2 sockets
Sep 27 05:06:49 dist-test-slave-hgs0 krb5kdc[8653](info): commencing operation
krb5kdc: starting...
W20260927 05:06:50.949361 388 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:06:51 dist-test-slave-hgs0 krb5kdc[8653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:06:51.206544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45217
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:51.557978 8666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:51.558306 8666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:51.558473 8666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:51.566102 8666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:51.566237 8666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:51.566331 8666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:51.566449 8666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:51.566538 8666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:51.578231 8666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:51.579684 8666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:51.581172 8666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:51.593724 8674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:51.593782 8672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:51.593781 8671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:51.594348 8666 server_base.cc:1034] running on GCE node
I20260927 05:06:51.595299 8666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:51.596971 8666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:51.598213 8666 hybrid_clock.cc:648] HybridClock initialized: now 1790485611598154 us; error 58 us; skew 500 ppm
I20260927 05:06:51.598598 8666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:51 dist-test-slave-hgs0 krb5kdc[8653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:51.606420 8666 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:06:51.608589 8666 webserver.cc:460] Webserver started at http://127.0.97.62:39485/ using document root <none> and password file <none>
I20260927 05:06:51.609604 8666 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:51.609787 8666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:51.610189 8666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:51.613898 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "cb5b5c9549af436498cc56549d3d1716"
format_stamp: "Formatted at 2026-09-27 05:06:51 on dist-test-slave-hgs0"
server_key: "2ee8c4b4795714ed6109256d2c990da4"
server_key_iv: "ee6fb3290328d3ea8a8b49b11a919e3a"
server_key_version: "encryptionkey@0"
I20260927 05:06:51.614787 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "cb5b5c9549af436498cc56549d3d1716"
format_stamp: "Formatted at 2026-09-27 05:06:51 on dist-test-slave-hgs0"
server_key: "2ee8c4b4795714ed6109256d2c990da4"
server_key_iv: "ee6fb3290328d3ea8a8b49b11a919e3a"
server_key_version: "encryptionkey@0"
I20260927 05:06:51.620625 8666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:06:51.624491 8681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:51.626027 8666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:06:51.626255 8666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "cb5b5c9549af436498cc56549d3d1716"
format_stamp: "Formatted at 2026-09-27 05:06:51 on dist-test-slave-hgs0"
server_key: "2ee8c4b4795714ed6109256d2c990da4"
server_key_iv: "ee6fb3290328d3ea8a8b49b11a919e3a"
server_key_version: "encryptionkey@0"
I20260927 05:06:51.626477 8666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:51.663892 8666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:51.668298 8666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:51.668596 8666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:51.687175 8666 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45217
I20260927 05:06:51.687204 8732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45217 every 8 connection(s)
I20260927 05:06:51.688720 8666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:51.691026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 8666
I20260927 05:06:51.691260 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:06:51.691883 388 external_mini_cluster.cc:1428] Setting key 04c2ee9e537d3ec74b230f4706b3278e
I20260927 05:06:51.692978 8733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 05:06:51 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:51.710161 8733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716: Bootstrap starting.
I20260927 05:06:51.713935 8733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716: Neither blocks nor log segments found. Creating new log.
I20260927 05:06:51.715636 8733 log.cc:824] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:51.718221 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:51.694054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.0.97.62:45217)
0927 05:06:51.694889 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:51.694902 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:51.694944 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:06:51.696586 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:51.696610 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:51.697118 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:51.697506 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:51.699476 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:51.701206 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:51.703307 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:51.704288 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:51.704456 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:51.710050 (+ 5594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:51.710061 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:51.710080 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:51.710127 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:51.713487 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:51.714470 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:51.714480 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:51.714488 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:51.714663 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:51.715086 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:51.715089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:51.715091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:51.715653 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:51.715929 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:51.716542 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 05:06:51.717127 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":365,"threads_started":1}
I20260927 05:06:51.719092 8733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716: No bootstrap required, opened a new log
I20260927 05:06:51.727821 8733 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } }
I20260927 05:06:51.728435 8733 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:06:51.728582 8733 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5b5c9549af436498cc56549d3d1716, State: Initialized, Role: FOLLOWER
I20260927 05:06:51.729255 8733 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } }
I20260927 05:06:51.729488 8733 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:51.729621 8733 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:51.729768 8733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:06:51.732792 8733 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } }
I20260927 05:06:51.733469 8733 leader_election.cc:302] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5b5c9549af436498cc56549d3d1716; no voters:
I20260927 05:06:51.734081 8733 leader_election.cc:288] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:06:51.734247 8738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:06:51.734953 8738 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 1 LEADER]: Becoming Leader. State: Replica: cb5b5c9549af436498cc56549d3d1716, State: Running, Role: LEADER
I20260927 05:06:51.735610 8738 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } }
I20260927 05:06:51.736389 8733 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:51.738528 8740 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5b5c9549af436498cc56549d3d1716. Latest consensus state: current_term: 1 leader_uuid: "cb5b5c9549af436498cc56549d3d1716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } } }
I20260927 05:06:51.738898 8740 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:51.739166 8739 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb5b5c9549af436498cc56549d3d1716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } } }
I20260927 05:06:51.739609 8739 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:51.748490 8744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:51.755050 8744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:51.765558 8744 catalog_manager.cc:1347] Generated new cluster ID: da151d7b61e74af2aab0cf50c2d7f972
I20260927 05:06:51.765758 8744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:51.784138 8744 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:06:51.785763 8744 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:51.794554 8744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716: Generated new TSK 0
I20260927 05:06:51.795908 8744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:06:51.860049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45217
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:52.242686 8759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:52.243074 8759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:52.243261 8759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:52.250823 8759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:52.250957 8759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:52.251149 8759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:52.261844 8759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:52.263433 8759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:52.265014 8759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:52.279510 8765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:52.279459 8767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:52.279541 8764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:52.280704 8759 server_base.cc:1034] running on GCE node
I20260927 05:06:52.281512 8759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:52.283217 8759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:52.284484 8759 hybrid_clock.cc:648] HybridClock initialized: now 1790485612284423 us; error 44 us; skew 500 ppm
I20260927 05:06:52.284878 8759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:52 dist-test-slave-hgs0 krb5kdc[8653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:52.288602 8759 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:06:52.290493 8759 webserver.cc:460] Webserver started at http://127.0.97.1:44841/ using document root <none> and password file <none>
I20260927 05:06:52.291361 8759 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:52.291531 8759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:52.291919 8759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:52.295792 8759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "10cdcdabebdc4cf58808454701f48a57"
format_stamp: "Formatted at 2026-09-27 05:06:52 on dist-test-slave-hgs0"
server_key: "9a9bcb0dd23db51a4f417183a603254a"
server_key_iv: "0b4953d67979546261eddb604bf8f1a5"
server_key_version: "encryptionkey@0"
I20260927 05:06:52.296830 8759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "10cdcdabebdc4cf58808454701f48a57"
format_stamp: "Formatted at 2026-09-27 05:06:52 on dist-test-slave-hgs0"
server_key: "9a9bcb0dd23db51a4f417183a603254a"
server_key_iv: "0b4953d67979546261eddb604bf8f1a5"
server_key_version: "encryptionkey@0"
I20260927 05:06:52.302795 8759 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:06:52.307015 8774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:52.309131 8759 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:06:52.309358 8759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "10cdcdabebdc4cf58808454701f48a57"
format_stamp: "Formatted at 2026-09-27 05:06:52 on dist-test-slave-hgs0"
server_key: "9a9bcb0dd23db51a4f417183a603254a"
server_key_iv: "0b4953d67979546261eddb604bf8f1a5"
server_key_version: "encryptionkey@0"
I20260927 05:06:52.309634 8759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:52.342281 8759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:52.346952 8759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:52.347270 8759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:52.348351 8759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:52.349786 8759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:52.349862 8759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:52.350041 8759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:52.350104 8759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:52.383982 8759 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33821
I20260927 05:06:52.384007 8886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33821 every 8 connection(s)
I20260927 05:06:52.385382 8759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:52.395682 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 8759
I20260927 05:06:52.395905 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:06:52.396656 388 external_mini_cluster.cc:1428] Setting key b0b1e127f8179f30656b5ba98c290f60
Sep 27 05:06:52 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:52.406339 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:52.388150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49301 (local address 127.0.97.62:45217)
0927 05:06:52.388577 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:52.388581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:52.389547 (+ 966us) server_negotiation.cc:407] Connection header received
0927 05:06:52.391401 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:52.391407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:52.391508 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:52.391626 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:52.393252 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:52.393968 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:52.395820 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:52.396327 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:52.396435 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:52.400442 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:52.400447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:52.400451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:52.400482 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:52.403106 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:52.403911 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:52.403922 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:52.403925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:52.404096 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:52.404551 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:52.404556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:52.404559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:52.404827 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:52.404986 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:52.405755 (+ 769us) server_negotiation.cc:299] Negotiation successful
0927 05:06:52.405979 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":223,"threads_started":1}
I20260927 05:06:52.407392 8890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:52.388681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:49301)
0927 05:06:52.389385 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:52.389424 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:52.391117 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:52.391880 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:52.391889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:52.392474 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:52.393074 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:52.393092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:52.394120 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:52.394125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:52.395587 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:52.395621 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:52.396570 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:52.396578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:52.396908 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:52.397719 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:52.397747 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:52.400219 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:52.403285 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:52.403296 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:52.403316 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:52.403771 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:52.404213 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:52.404218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:52.404222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:52.404421 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:52.404960 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:52.404969 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:52.405347 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 05:06:52.405624 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 05:06:52.406213 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":311,"threads_started":1}
I20260927 05:06:52.409093 8887 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45217
I20260927 05:06:52.409403 8887 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:52.410168 8887 heartbeater.cc:511] Master 127.0.97.62:45217 requested a full tablet report, sending...
I20260927 05:06:52.412449 8698 ts_manager.cc:194] Registered new tserver with Master: 10cdcdabebdc4cf58808454701f48a57 (127.0.97.1:33821)
I20260927 05:06:52.414186 8698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49301
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:06:52.448863 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45217
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:52.810883 8893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:52.811266 8893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:52.811455 8893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:52.819644 8893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:52.819834 8893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:52.820065 8893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:52.832906 8893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:52.834586 8893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:52.836059 8893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:52.849748 8901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:52.849768 8899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:52.850381 8893 server_base.cc:1034] running on GCE node
W20260927 05:06:52.850613 8898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:52.851380 8893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:52.852504 8893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:52.853739 8893 hybrid_clock.cc:648] HybridClock initialized: now 1790485612853679 us; error 67 us; skew 500 ppm
I20260927 05:06:52.854074 8893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:52 dist-test-slave-hgs0 krb5kdc[8653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:52.857298 8893 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:06:52.858868 8893 webserver.cc:460] Webserver started at http://127.0.97.2:45753/ using document root <none> and password file <none>
I20260927 05:06:52.859619 8893 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:52.859769 8893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:52.860178 8893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:52.863142 8893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "04a4f9f7a8a04835a21a583957e12756"
format_stamp: "Formatted at 2026-09-27 05:06:52 on dist-test-slave-hgs0"
server_key: "b930dba45ea1e513b5205aef5806d223"
server_key_iv: "7752b616f58b76980ef6bcb285031ce3"
server_key_version: "encryptionkey@0"
I20260927 05:06:52.863929 8893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "04a4f9f7a8a04835a21a583957e12756"
format_stamp: "Formatted at 2026-09-27 05:06:52 on dist-test-slave-hgs0"
server_key: "b930dba45ea1e513b5205aef5806d223"
server_key_iv: "7752b616f58b76980ef6bcb285031ce3"
server_key_version: "encryptionkey@0"
I20260927 05:06:52.869316 8893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:06:52.873072 8908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:52.874680 8893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:06:52.874871 8893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "04a4f9f7a8a04835a21a583957e12756"
format_stamp: "Formatted at 2026-09-27 05:06:52 on dist-test-slave-hgs0"
server_key: "b930dba45ea1e513b5205aef5806d223"
server_key_iv: "7752b616f58b76980ef6bcb285031ce3"
server_key_version: "encryptionkey@0"
I20260927 05:06:52.875092 8893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:52.902395 8893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:52.906652 8893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:52.906971 8893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:52.908043 8893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:52.909437 8893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:52.909555 8893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:52.909729 8893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:52.909804 8893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:52.943984 8893 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43111
I20260927 05:06:52.944049 9020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43111 every 8 connection(s)
I20260927 05:06:52.945281 8893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:52.953459 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 8893
I20260927 05:06:52.953652 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:06:52.954101 388 external_mini_cluster.cc:1428] Setting key 931af18e748bcf399f0a70c5722cf809
Sep 27 05:06:52 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:52.967231 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:52.947877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51555 (local address 127.0.97.62:45217)
0927 05:06:52.948298 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:52.948302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:52.949336 (+ 1034us) server_negotiation.cc:407] Connection header received
0927 05:06:52.951503 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:52.951509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:52.951614 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:52.951740 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:52.953479 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:52.954407 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:52.956539 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:52.956966 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:52.957074 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:52.961356 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:52.961362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:52.961366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:52.961399 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:52.964026 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:52.964794 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:52.964799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:52.964805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:52.964932 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:52.965387 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:52.965396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:52.965398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:52.965681 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:52.965838 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:52.966628 (+ 790us) server_negotiation.cc:299] Negotiation successful
0927 05:06:52.966859 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":223,"threads_started":1}
I20260927 05:06:52.968375 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:52.948503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:51555)
0927 05:06:52.949175 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:52.949218 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:52.951116 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:52.952043 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:52.952051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:52.952582 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:52.953269 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:52.953297 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:52.954573 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:52.954578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:52.956382 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:52.956397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:52.957210 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:52.957220 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:52.957581 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:52.958328 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:52.958361 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:52.961091 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:52.964176 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:52.964186 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:52.964206 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:52.964657 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:52.965056 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:52.965061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:52.965065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:52.965261 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:52.965821 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:52.965829 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:52.966205 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 05:06:52.966504 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 05:06:52.967090 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":262,"threads_started":1}
I20260927 05:06:52.969944 9021 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45217
I20260927 05:06:52.970304 9021 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:52.971251 9021 heartbeater.cc:511] Master 127.0.97.62:45217 requested a full tablet report, sending...
I20260927 05:06:52.973162 8698 ts_manager.cc:194] Registered new tserver with Master: 04a4f9f7a8a04835a21a583957e12756 (127.0.97.2:43111)
I20260927 05:06:52.974512 8698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51555
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:06:53.012501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45217
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:53.377076 9027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:53.377444 9027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:53.377632 9027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:53.385231 9027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:53.385389 9027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:53.385578 9027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:53.396086 9027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:53.397485 9027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:53.398867 9027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:53.412559 9035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:53.412647 9032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:53.412647 9033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:53.413551 9027 server_base.cc:1034] running on GCE node
I20260927 05:06:53.414196 9027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:53.415244 9027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:53.416441 9027 hybrid_clock.cc:648] HybridClock initialized: now 1790485613416397 us; error 36 us; skew 500 ppm
I20260927 05:06:53.416824 9027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:06:53.417672 8887 heartbeater.cc:503] Master 127.0.97.62:45217 was elected leader, sending a full tablet report...
Sep 27 05:06:53 dist-test-slave-hgs0 krb5kdc[8653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:53.420393 9027 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:06:53.421838 9027 webserver.cc:460] Webserver started at http://127.0.97.3:40157/ using document root <none> and password file <none>
I20260927 05:06:53.422559 9027 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:53.422704 9027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:53.423020 9027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:06:53.426179 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "a65fcc8e1a2b471c974cb149e728b0f2"
format_stamp: "Formatted at 2026-09-27 05:06:53 on dist-test-slave-hgs0"
server_key: "72564006f21a95100b11ad05cfe80898"
server_key_iv: "7b7833d614240897ba67f7b32232afa2"
server_key_version: "encryptionkey@0"
I20260927 05:06:53.426975 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a65fcc8e1a2b471c974cb149e728b0f2"
format_stamp: "Formatted at 2026-09-27 05:06:53 on dist-test-slave-hgs0"
server_key: "72564006f21a95100b11ad05cfe80898"
server_key_iv: "7b7833d614240897ba67f7b32232afa2"
server_key_version: "encryptionkey@0"
I20260927 05:06:53.432367 9027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 05:06:53.435899 9042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:53.437544 9027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:06:53.437726 9027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "a65fcc8e1a2b471c974cb149e728b0f2"
format_stamp: "Formatted at 2026-09-27 05:06:53 on dist-test-slave-hgs0"
server_key: "72564006f21a95100b11ad05cfe80898"
server_key_iv: "7b7833d614240897ba67f7b32232afa2"
server_key_version: "encryptionkey@0"
I20260927 05:06:53.437937 9027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:53.456482 9027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:53.460974 9027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:53.461282 9027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:53.462363 9027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:53.463920 9027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:53.464077 9027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 05:06:53.464241 9027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:53.464309 9027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:53.500825 9027 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39241
I20260927 05:06:53.500916 9154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39241 every 8 connection(s)
I20260927 05:06:53.502255 9027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:53.510032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 9027
I20260927 05:06:53.510183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:06:53.510807 388 external_mini_cluster.cc:1428] Setting key 587c6a2cd830bf3a213b872fe5c222b2
Sep 27 05:06:53 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:06:53.525519 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:53.505136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49615 (local address 127.0.97.62:45217)
0927 05:06:53.505636 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:53.505643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:53.506848 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 05:06:53.508956 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:53.508960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:53.509062 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:53.509185 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:53.511387 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:53.512333 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:53.514992 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:53.515447 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:53.515564 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:53.519504 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:53.519510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:53.519513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:53.519550 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:53.522214 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:53.523024 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:53.523028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:53.523030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:53.523144 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:53.523539 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:53.523543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:53.523545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:53.523807 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:53.523919 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:53.524888 (+ 969us) server_negotiation.cc:299] Negotiation successful
0927 05:06:53.525150 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":223,"threads_started":1}
I20260927 05:06:53.526625 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:53.505803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:49615)
0927 05:06:53.506648 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:53.506699 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:53.508630 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:53.509502 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:53.509520 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:53.510211 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:53.511142 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:53.511171 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:53.512532 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:53.512537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:53.514842 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:53.514859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:53.515679 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:53.515684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:53.515935 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:53.516901 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:53.516945 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:53.519263 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:06:53.522393 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:53.522400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:53.522417 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:53.522893 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:53.523242 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:06:53.523245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:06:53.523248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:06:53.523430 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:06:53.524051 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:06:53.524059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:06:53.524464 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 05:06:53.524749 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 05:06:53.525324 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":367,"threads_started":1}
I20260927 05:06:53.528174 9155 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45217
I20260927 05:06:53.528456 9155 heartbeater.cc:464] Registering TS with master...
I20260927 05:06:53.529186 9155 heartbeater.cc:511] Master 127.0.97.62:45217 requested a full tablet report, sending...
I20260927 05:06:53.530689 8698 ts_manager.cc:194] Registered new tserver with Master: a65fcc8e1a2b471c974cb149e728b0f2 (127.0.97.3:39241)
I20260927 05:06:53.531591 8698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49615
I20260927 05:06:53.539776 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260927 05:06:53.691730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 8759
Sep 27 05:06:53 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:06:53.710752 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:53.693090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.0.97.1:33821)
0927 05:06:53.693883 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:53.693890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:53.693918 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:06:53.693978 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:53.693985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:53.694247 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:53.694425 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:53.695840 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:53.697169 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:53.698647 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:53.699329 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:53.699497 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:53.704436 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:53.704452 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:53.704470 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:53.704534 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:53.707738 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:53.708571 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:53.708581 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:53.708590 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:53.708726 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:53.709113 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:53.709118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:53.709121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:53.709487 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:53.709810 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:53.710112 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 05:06:53.710391 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":413,"threads_started":1}
I20260927 05:06:53.712176 8882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:53.800791 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 8759
I20260927 05:06:53.817685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 8893
Sep 27 05:06:53 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:06:53.834213 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:53.818709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.0.97.2:43111)
0927 05:06:53.819309 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:53.819318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:53.819345 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:06:53.819408 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:53.819415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:53.819661 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:53.819853 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:53.821260 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:53.822365 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:53.823711 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:53.824379 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:53.824530 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:53.828785 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:53.828798 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:53.828813 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:53.828848 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:53.831742 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:53.832458 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:53.832465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:53.832468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:53.832589 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:53.832900 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:53.832903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:53.832905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:53.833263 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:53.833490 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:53.833706 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 05:06:53.833906 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":304,"threads_started":1}
I20260927 05:06:53.835491 9016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:53.920158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 8893
I20260927 05:06:53.937093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9027
Sep 27 05:06:53 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:06:53.954202 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:53.938025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.0.97.3:39241)
0927 05:06:53.938524 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:53.938533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:53.938565 (+ 32us) server_negotiation.cc:407] Connection header received
0927 05:06:53.938628 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:53.938634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:53.938870 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:53.939008 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:53.940283 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:53.941301 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:53.942937 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:53.943586 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:53.943731 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:53.948221 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:53.948234 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:53.948252 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:53.948291 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:53.951041 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:53.951884 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:53.951892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:53.951894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:53.952104 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:53.952495 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:53.952518 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:53.952522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:53.952896 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:53.953234 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:53.953532 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 05:06:53.953809 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":234,"threads_started":1}
I20260927 05:06:53.955533 9150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:54.043049 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 9027
I20260927 05:06:54.060223 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 8666
I20260927 05:06:54.061409 8728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:06:54.172207 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 8666
I20260927 05:06:54.195480 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45217
--webserver_interface=127.0.97.62
--webserver_port=39485
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:06:54.539240 9173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:54.539541 9173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:54.539629 9173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:54.546937 9173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:06:54.547070 9173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:54.547117 9173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:54.547223 9173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:06:54.547284 9173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:06:54.557479 9173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=39485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:54.558908 9173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:54.560279 9173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:54.572420 9179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:54.572783 9178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:54.572837 9181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:54.573688 9173 server_base.cc:1034] running on GCE node
I20260927 05:06:54.574472 9173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:54.575985 9173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:54.577222 9173 hybrid_clock.cc:648] HybridClock initialized: now 1790485614577169 us; error 40 us; skew 500 ppm
I20260927 05:06:54.577553 9173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:54 dist-test-slave-hgs0 krb5kdc[8653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:54.580788 9173 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:06:54.582278 9173 webserver.cc:460] Webserver started at http://127.0.97.62:39485/ using document root <none> and password file <none>
I20260927 05:06:54.582964 9173 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:54.583086 9173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:54.589030 9173 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:06:54.592360 9188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:54.594036 9173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 05:06:54.594204 9173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "cb5b5c9549af436498cc56549d3d1716"
format_stamp: "Formatted at 2026-09-27 05:06:51 on dist-test-slave-hgs0"
server_key: "2ee8c4b4795714ed6109256d2c990da4"
server_key_iv: "ee6fb3290328d3ea8a8b49b11a919e3a"
server_key_version: "encryptionkey@0"
I20260927 05:06:54.595259 9173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:54.618848 9173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:54.623342 9173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:54.623574 9173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:54.643491 9173 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45217
I20260927 05:06:54.643510 9239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45217 every 8 connection(s)
I20260927 05:06:54.645248 9173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:06:54.649560 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 9173
I20260927 05:06:54.650452 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33821
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45217
--webserver_port=44841
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:06:54.653748 9240 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:06:54.657636 9240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716: Bootstrap starting.
I20260927 05:06:54.686901 9240 log.cc:824] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716: Log is configured to *not* fsync() on all Append() calls
I20260927 05:06:54.699905 9240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:06:54.700738 9240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716: Bootstrap complete.
I20260927 05:06:54.715184 9240 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } }
I20260927 05:06:54.716637 9240 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5b5c9549af436498cc56549d3d1716, State: Initialized, Role: FOLLOWER
I20260927 05:06:54.717501 9240 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } }
I20260927 05:06:54.717818 9240 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:06:54.717952 9240 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:06:54.718148 9240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:06:54.725286 9240 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } }
I20260927 05:06:54.726150 9240 leader_election.cc:302] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5b5c9549af436498cc56549d3d1716; no voters:
I20260927 05:06:54.726850 9240 leader_election.cc:288] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:06:54.727027 9244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:06:54.728252 9244 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [term 2 LEADER]: Becoming Leader. State: Replica: cb5b5c9549af436498cc56549d3d1716, State: Running, Role: LEADER
I20260927 05:06:54.728838 9244 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } }
I20260927 05:06:54.729161 9240 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:06:54.731400 9246 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5b5c9549af436498cc56549d3d1716. Latest consensus state: current_term: 2 leader_uuid: "cb5b5c9549af436498cc56549d3d1716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } } }
I20260927 05:06:54.731798 9246 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:54.732388 9245 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cb5b5c9549af436498cc56549d3d1716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5b5c9549af436498cc56549d3d1716" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45217 } } }
I20260927 05:06:54.732712 9245 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716 [sys.catalog]: This master's current role is: LEADER
I20260927 05:06:54.738435 9251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:06:54.745236 9251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:06:54.746824 9251 catalog_manager.cc:1259] Loaded cluster ID: da151d7b61e74af2aab0cf50c2d7f972
I20260927 05:06:54.746946 9251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:06:54.751202 9251 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:06:54.752883 9251 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cb5b5c9549af436498cc56549d3d1716: Loaded TSK: 0
I20260927 05:06:54.753496 9261 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 05:06:54.754777 9251 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:06:55.067746 9242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:55.068192 9242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:55.068352 9242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:55.076380 9242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:55.076575 9242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:55.076790 9242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:06:55.087893 9242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44841
--tserver_master_addrs=127.0.97.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:55.089502 9242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:55.090915 9242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:55.107620 9266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:55.107568 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:55.109287 9269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:55.113893 9242 server_base.cc:1034] running on GCE node
I20260927 05:06:55.114741 9242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:55.116150 9242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:55.117440 9242 hybrid_clock.cc:648] HybridClock initialized: now 1790485615117379 us; error 60 us; skew 500 ppm
I20260927 05:06:55.117869 9242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485615, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:55.122366 9242 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260927 05:06:55.124536 9242 webserver.cc:460] Webserver started at http://127.0.97.1:44841/ using document root <none> and password file <none>
I20260927 05:06:55.125409 9242 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:55.125608 9242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:55.132455 9242 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:06:55.135618 9276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:55.137383 9242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:06:55.137606 9242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "10cdcdabebdc4cf58808454701f48a57"
format_stamp: "Formatted at 2026-09-27 05:06:52 on dist-test-slave-hgs0"
server_key: "9a9bcb0dd23db51a4f417183a603254a"
server_key_iv: "0b4953d67979546261eddb604bf8f1a5"
server_key_version: "encryptionkey@0"
I20260927 05:06:55.138751 9242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:55.165971 9242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:55.170898 9242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:55.171149 9242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:55.172258 9242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:55.173820 9242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:55.173924 9242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:55.174006 9242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:55.174065 9242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:55.210312 9242 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33821
I20260927 05:06:55.210435 9388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33821 every 8 connection(s)
I20260927 05:06:55.212126 9242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:06:55.221786 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 9242
I20260927 05:06:55.222506 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:43111
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45217
--webserver_port=45753
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:55.233625 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:55.215874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:47233)
0927 05:06:55.216869 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:55.216913 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:55.218911 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:55.220711 (+ 1800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:55.220724 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:55.221462 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:55.222238 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.222262 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.224139 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.224144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.225530 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.225545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.226739 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.226746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.227041 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.227827 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:55.227857 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:55.232317 (+ 4460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":408,"threads_started":1}
W20260927 05:06:55.234202 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:55.215699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47233 (local address 127.0.97.62:45217)
0927 05:06:55.216636 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:55.216655 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:55.217076 (+ 421us) server_negotiation.cc:407] Connection header received
0927 05:06:55.219542 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:55.219556 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:55.220045 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:55.220485 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:55.222487 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.223897 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.225729 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.226451 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.226690 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.232825 (+ 6135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47233: BlockingRecv error: recv got EOF from 127.0.97.1:47233 (error 108)
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":330,"threads_started":1}
W20260927 05:06:55.234462 9389 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45217 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45217: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:55.244867 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:55.235193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:46031)
0927 05:06:55.235367 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:55.235390 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:55.235553 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:55.235924 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:55.235930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:55.236376 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:55.236630 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.236641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.237853 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.237859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.239134 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.239149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.239922 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.239928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.240158 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.240859 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:55.240879 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:55.244453 (+ 3574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:06:55.244980 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:55.235391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46031 (local address 127.0.97.62:45217)
0927 05:06:55.235597 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:55.235607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:55.235632 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:06:55.235700 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:55.235706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:55.235778 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:55.235894 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:55.236802 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.237696 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.239310 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.239745 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.239894 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.244739 (+ 4845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46031: BlockingRecv error: recv got EOF from 127.0.97.1:46031 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:55.255595 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:55.245805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:59061)
0927 05:06:55.246056 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:55.246093 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:55.246252 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:55.246608 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:55.246616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:55.247116 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:55.247380 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.247398 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.248582 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.248588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.250210 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.250228 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.251029 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.251035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.251259 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.251912 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:55.251962 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:55.255357 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 05:06:55.255896 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:55.245928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59061 (local address 127.0.97.62:45217)
0927 05:06:55.246082 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:55.246086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:55.246179 (+ 93us) server_negotiation.cc:407] Connection header received
0927 05:06:55.246380 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:55.246384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:55.246445 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:55.246529 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:55.247551 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.248419 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.250380 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.250870 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.250978 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.255646 (+ 4668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59061: BlockingRecv error: recv got EOF from 127.0.97.1:59061 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 05:06:55.256047 9389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:06:55.573156 9393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:55.573455 9393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:55.573624 9393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:55.581097 9393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:55.581223 9393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:55.581432 9393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:06:55.592027 9393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:43111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45753
--tserver_master_addrs=127.0.97.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:55.593626 9393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:55.595117 9393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:55.609927 9398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:55.610030 9399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:55.609997 9401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:55.611415 9393 server_base.cc:1034] running on GCE node
I20260927 05:06:55.612249 9393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:55.613390 9393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:55.614652 9393 hybrid_clock.cc:648] HybridClock initialized: now 1790485615614607 us; error 56 us; skew 500 ppm
I20260927 05:06:55.615038 9393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485615, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:55.618584 9393 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260927 05:06:55.620210 9393 webserver.cc:460] Webserver started at http://127.0.97.2:45753/ using document root <none> and password file <none>
I20260927 05:06:55.620973 9393 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:55.621150 9393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:55.627300 9393 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:06:55.630767 9408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:55.632405 9393 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:06:55.632642 9393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "04a4f9f7a8a04835a21a583957e12756"
format_stamp: "Formatted at 2026-09-27 05:06:52 on dist-test-slave-hgs0"
server_key: "b930dba45ea1e513b5205aef5806d223"
server_key_iv: "7752b616f58b76980ef6bcb285031ce3"
server_key_version: "encryptionkey@0"
I20260927 05:06:55.633740 9393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:55.654184 9393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:55.658726 9393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:55.659046 9393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:55.660211 9393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:55.661623 9393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:55.661749 9393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:55.661885 9393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:55.661985 9393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:55.697633 9393 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43111
I20260927 05:06:55.697738 9520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43111 every 8 connection(s)
I20260927 05:06:55.699430 9393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:06:55.709417 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 9393
I20260927 05:06:55.710107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39241
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45217
--webserver_port=40157
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:06:55.717649 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:55.702405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55393 (local address 127.0.97.62:45217)
0927 05:06:55.702637 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:55.702644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:55.704064 (+ 1420us) server_negotiation.cc:407] Connection header received
0927 05:06:55.705982 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:55.705986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:55.706056 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:55.706154 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:55.707789 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.708593 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.710526 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.711382 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.711553 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.717352 (+ 5799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55393: BlockingRecv error: recv got EOF from 127.0.97.2:55393 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 05:06:55.718494 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:55.703058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:55393)
0927 05:06:55.703840 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:55.703883 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:55.705666 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:55.706395 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:55.706404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:55.706941 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:55.707614 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.707634 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.708772 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.708777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.710243 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.710288 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.711685 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.711692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.712034 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.712860 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:55.712893 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:55.717098 (+ 4205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":316,"threads_started":1}
W20260927 05:06:55.719465 9521 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45217 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45217: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:55.730062 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:55.720247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:35075)
0927 05:06:55.720460 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:55.720484 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:55.720630 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:55.721066 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:55.721073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:55.721483 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:55.721654 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.721664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.723023 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.723032 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.724116 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.724131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.725176 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.725184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.725384 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.726093 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:55.726113 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:55.729641 (+ 3528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 05:06:55.730859 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:55.720385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35075 (local address 127.0.97.62:45217)
0927 05:06:55.720623 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:55.720630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:55.720653 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:06:55.720778 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:55.720785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:55.720870 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:55.721041 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:55.721851 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.722847 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.724288 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.724992 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.725158 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.729964 (+ 4806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35075: BlockingRecv error: recv got EOF from 127.0.97.2:35075 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:55 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:55.739975 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:55.730934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:58237)
0927 05:06:55.731260 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:55.731284 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:55.731403 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:55.731850 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:55.731856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:55.732215 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:55.732381 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.732390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.733501 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.733514 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.734483 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:55.734495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.735279 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.735286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:55.735486 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.736186 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:55.736205 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:55.739546 (+ 3341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183}
W20260927 05:06:55.739913 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:55.731167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58237 (local address 127.0.97.62:45217)
0927 05:06:55.731454 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:55.731464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:55.731493 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:06:55.731588 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:55.731597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:55.731697 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:55.731828 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:55.732564 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.733337 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.734649 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:55.735117 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:55.735266 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:55.739728 (+ 4462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58237: BlockingRecv error: recv got EOF from 127.0.97.2:58237 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":448,"threads_started":1}
W20260927 05:06:55.740285 9521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:06:56.058797 9524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:06:56.059100 9524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:06:56.059270 9524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:06:56.066778 9524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:06:56.066910 9524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:06:56.067095 9524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:06:56.078112 9524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40157
--tserver_master_addrs=127.0.97.62:45217
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:06:56.079713 9524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:06:56.081283 9524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:06:56.096269 9533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:56.096182 9530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:06:56.096297 9531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:06:56.097141 9524 server_base.cc:1034] running on GCE node
I20260927 05:06:56.097900 9524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:06:56.099023 9524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:06:56.100270 9524 hybrid_clock.cc:648] HybridClock initialized: now 1790485616100215 us; error 47 us; skew 500 ppm
I20260927 05:06:56.100651 9524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485616, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:06:56.104135 9524 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260927 05:06:56.105837 9524 webserver.cc:460] Webserver started at http://127.0.97.3:40157/ using document root <none> and password file <none>
I20260927 05:06:56.106577 9524 fs_manager.cc:359] Metadata directory not provided
I20260927 05:06:56.106698 9524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:06:56.113832 9524 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:06:56.117050 9540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:56.118677 9524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:06:56.118861 9524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "a65fcc8e1a2b471c974cb149e728b0f2"
format_stamp: "Formatted at 2026-09-27 05:06:53 on dist-test-slave-hgs0"
server_key: "72564006f21a95100b11ad05cfe80898"
server_key_iv: "7b7833d614240897ba67f7b32232afa2"
server_key_version: "encryptionkey@0"
I20260927 05:06:56.119881 9524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:06:56.142618 9524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:06:56.147902 9524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:06:56.148299 9524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:06:56.149398 9524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:06:56.150861 9524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:06:56.150956 9524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:56.151038 9524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:06:56.151093 9524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:06:56.186334 9524 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39241
I20260927 05:06:56.186362 9652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39241 every 8 connection(s)
I20260927 05:06:56.188149 9524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:06:56.196830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 9524
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 05:06:56.207111 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:56.191229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34635 (local address 127.0.97.62:45217)
0927 05:06:56.191455 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:56.191462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:56.192770 (+ 1308us) server_negotiation.cc:407] Connection header received
0927 05:06:56.194641 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:56.194645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:56.194718 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:56.194811 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:56.196566 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.197624 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.200293 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.200770 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.200941 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.206714 (+ 5773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34635: BlockingRecv error: recv got EOF from 127.0.97.3:34635 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":80}
I20260927 05:06:56.207839 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:56.191750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:34635)
0927 05:06:56.192585 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:56.192633 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:56.194344 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:56.195051 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:56.195061 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:56.195626 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:56.196348 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:56.196368 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.197803 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.197810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:56.200116 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:56.200137 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.201039 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.201047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:56.201343 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.202109 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:56.202139 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:56.206410 (+ 4271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":363,"threads_started":1}
W20260927 05:06:56.208706 9653 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45217 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:45217: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260927 05:06:56.216118 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:56.197935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.0.97.62:45217)
0927 05:06:56.198196 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:56.198201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:56.198546 (+ 345us) server_negotiation.cc:407] Connection header received
0927 05:06:56.198708 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:56.198712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:56.198788 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:56.198884 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:56.200904 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.201818 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.203114 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.203860 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.204047 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.207222 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:06:56.207236 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:06:56.207262 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:06:56.207318 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:06:56.211791 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:56.212647 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:56.212660 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:56.212672 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:56.212971 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:06:56.213525 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:06:56.213533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:06:56.213538 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:06:56.214446 (+ 908us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:06:56.214821 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0927 05:06:56.215307 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 05:06:56.215731 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:56.220826 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:56.209418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:56645)
0927 05:06:56.209748 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:56.209772 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:56.209956 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:56.210400 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:56.210406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:56.210865 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:56.211074 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:56.211087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.212716 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.212723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:56.214378 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:56.214397 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.215805 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.215812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:56.216056 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.216734 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:56.216757 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:56.220599 (+ 3842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191}
W20260927 05:06:56.221010 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:56.209538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56645 (local address 127.0.97.62:45217)
0927 05:06:56.209749 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:56.209755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:56.209847 (+ 92us) server_negotiation.cc:407] Connection header received
0927 05:06:56.210131 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:56.210142 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:56.210224 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:56.210335 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:56.211275 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.212541 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.214529 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.215251 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.215387 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.220805 (+ 5418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56645: BlockingRecv error: recv got EOF from 127.0.97.3:56645 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:56.229233 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:56.221575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:39599)
0927 05:06:56.221744 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:56.221765 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:56.221853 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:56.222163 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:56.222166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:56.222435 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:56.222583 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:56.222591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.223699 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.223703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:56.224759 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:56.224771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.225517 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.225521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:56.225648 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.226241 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:56.226258 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:56.229008 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 05:06:56.229547 9653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:06:56.229558 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:56.221620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39599 (local address 127.0.97.62:45217)
0927 05:06:56.221831 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:56.221835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:56.221850 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:06:56.221949 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:56.221986 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:56.222043 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:56.222122 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:56.222700 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.223552 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.224893 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.225366 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.225471 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.229263 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39599: BlockingRecv error: recv got EOF from 127.0.97.3:39599 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:56.265044 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:56.256705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:41211)
0927 05:06:56.257398 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:56.257414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:56.257528 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:56.257814 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:56.257818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:56.258280 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:56.258465 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:56.258474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.259507 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.259511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:56.260546 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:56.260559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.261256 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.261260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:56.261379 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.261999 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:56.262017 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:56.264820 (+ 2803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":428,"threads_started":1}
W20260927 05:06:56.265277 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:56.256774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41211 (local address 127.0.97.62:45217)
0927 05:06:56.257016 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:56.257022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:56.257495 (+ 473us) server_negotiation.cc:407] Connection header received
0927 05:06:56.257623 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:56.257626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:56.257687 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:56.257766 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:56.258643 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.259371 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.260676 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.261103 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.261198 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.265029 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41211: BlockingRecv error: recv got EOF from 127.0.97.1:41211 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:56 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:56.753718 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:56.740989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:47563)
0927 05:06:56.741584 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:56.741602 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:56.741769 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:56.742140 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:56.742147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:56.742707 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:56.742906 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:56.742916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.744011 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.744017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:56.745068 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:56.745080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.745848 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.745854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:56.746040 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.750036 (+ 3996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:56.750057 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:56.753393 (+ 3336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":325,"threads_started":1}
W20260927 05:06:56.753894 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:56.741089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47563 (local address 127.0.97.62:45217)
0927 05:06:56.741327 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:56.741338 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:56.741697 (+ 359us) server_negotiation.cc:407] Connection header received
0927 05:06:56.741897 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:56.741903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:56.741974 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:56.742087 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:56.743062 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.743833 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.745219 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:56.745645 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:56.745800 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:56.753628 (+ 7828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47563: BlockingRecv error: recv got EOF from 127.0.97.2:47563 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 05:06:57 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:57 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:57.239681 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:57.230214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:47555)
0927 05:06:57.230775 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:57.230791 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:57.230901 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:57.231206 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:57.231209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:57.231642 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:57.231832 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:57.231840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.233046 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.233050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:57.234095 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:57.234109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.234861 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.234866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:57.234995 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:57.235694 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:57.235712 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:57.239380 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":278,"threads_started":1}
W20260927 05:06:57.239940 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:57.230309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47555 (local address 127.0.97.62:45217)
0927 05:06:57.230482 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:57.230486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:57.230868 (+ 382us) server_negotiation.cc:407] Connection header received
0927 05:06:57.231002 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:57.231005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:57.231065 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:57.231158 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:57.232081 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.232832 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.234260 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.234673 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.234776 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:57.239637 (+ 4861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47555: BlockingRecv error: recv got EOF from 127.0.97.3:47555 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 05:06:57 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:57 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:57.274148 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:57.266120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:34267)
0927 05:06:57.266640 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:57.266656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:57.266768 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:57.267025 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:57.267029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:57.267421 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:57.267574 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:57.267583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.268697 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.268701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:57.269647 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:57.269659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.270353 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.270357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:57.270476 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:57.271095 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:57.271113 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:57.273904 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":286,"threads_started":1}
W20260927 05:06:57.274513 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:57.266148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34267 (local address 127.0.97.62:45217)
0927 05:06:57.266304 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:57.266308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:57.266730 (+ 422us) server_negotiation.cc:407] Connection header received
0927 05:06:57.266863 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:57.266867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:57.266929 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:57.267008 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:57.267701 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.268545 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.269790 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.270208 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.270319 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:57.274181 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34267: BlockingRecv error: recv got EOF from 127.0.97.1:34267 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 05:06:57 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:57 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:57.764087 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:57.754841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:33039)
0927 05:06:57.755462 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:57.755481 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:57.755589 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:57.755999 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:57.756006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:57.756515 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:57.756696 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:57.756705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.757848 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.757852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:57.758950 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:57.758963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.759805 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.759810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:57.760009 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:57.760709 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:57.760729 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:57.763738 (+ 3009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":298,"threads_started":1}
W20260927 05:06:57.764264 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:57.755371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33039 (local address 127.0.97.62:45217)
0927 05:06:57.755584 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:57.755590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:57.755611 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:06:57.755702 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:57.755707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:57.755785 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:57.755890 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:57.756852 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.757694 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.759116 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:57.759646 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:57.759763 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:57.764007 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33039: BlockingRecv error: recv got EOF from 127.0.97.2:33039 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 05:06:58 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:58 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:58.250553 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:58.240804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:56017)
0927 05:06:58.241402 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:58.241419 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:58.241599 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:58.241959 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:58.241965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:58.242431 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:58.242598 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:58.242607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.243761 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.243767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:58.244881 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:58.244894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.245688 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.245709 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:58.245893 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:58.246603 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:58.246655 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:58.250187 (+ 3532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":311,"threads_started":1}
W20260927 05:06:58.250698 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:58.241029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56017 (local address 127.0.97.62:45217)
0927 05:06:58.241273 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:58.241280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:58.241507 (+ 227us) server_negotiation.cc:407] Connection header received
0927 05:06:58.241745 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:58.241750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:58.241814 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:58.241924 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:58.242758 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.243613 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.245051 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.245504 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.245668 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:58.250412 (+ 4744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56017: BlockingRecv error: recv got EOF from 127.0.97.3:56017 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 05:06:58 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:58 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:58.283222 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:58.275093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:37935)
0927 05:06:58.275652 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:58.275670 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:58.275789 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:58.276139 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:58.276142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:58.276513 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:58.276693 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:58.276702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.277688 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.277692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:58.278669 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:58.278682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.279315 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.279319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:58.279448 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:58.280087 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:58.280106 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:58.282919 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":308,"threads_started":1}
W20260927 05:06:58.283370 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:58.275186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37935 (local address 127.0.97.62:45217)
0927 05:06:58.275352 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:58.275357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:58.275748 (+ 391us) server_negotiation.cc:407] Connection header received
0927 05:06:58.275904 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:58.275907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:58.275981 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:58.276068 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:58.276837 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.277546 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.278792 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.279191 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.279279 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:58.283120 (+ 3841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37935: BlockingRecv error: recv got EOF from 127.0.97.1:37935 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 05:06:58 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:58 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:58.773844 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:58.765100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:39807)
0927 05:06:58.765663 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:58.765679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:58.765818 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:58.766126 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:58.766130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:58.766461 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:58.766631 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:58.766639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.767677 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.767680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:58.768775 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:58.768788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.769603 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.769607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:58.769744 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:58.770437 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:58.770459 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:58.773560 (+ 3101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":293,"threads_started":1}
W20260927 05:06:58.774049 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:58.765204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39807 (local address 127.0.97.62:45217)
0927 05:06:58.765409 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:58.765413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:58.765770 (+ 357us) server_negotiation.cc:407] Connection header received
0927 05:06:58.765918 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:58.765921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:58.765984 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:58.766069 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:58.766761 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.767555 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.768930 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:58.769442 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:58.769552 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:58.773799 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39807: BlockingRecv error: recv got EOF from 127.0.97.2:39807 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 05:06:59 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:59 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:59.260437 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:59.251906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:60083)
0927 05:06:59.252718 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:59.252747 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:59.252858 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:59.253253 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:59.253257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:59.253636 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:59.253858 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:59.253871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.254877 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.254886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:59.255809 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:59.255821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.256563 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.256568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:59.256755 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:59.257348 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:59.257365 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:59.260196 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":280,"threads_started":1}
W20260927 05:06:59.260674 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:59.252096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60083 (local address 127.0.97.62:45217)
0927 05:06:59.252365 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:59.252371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:59.252836 (+ 465us) server_negotiation.cc:407] Connection header received
0927 05:06:59.252979 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:59.252984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:59.253068 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:59.253229 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:59.254018 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.254744 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.255988 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.256408 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.256539 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:59.260432 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60083: BlockingRecv error: recv got EOF from 127.0.97.3:60083 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 27 05:06:59 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:59 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:59.293030 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:59.284377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:48701)
0927 05:06:59.284944 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:59.284960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:59.285167 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:59.285539 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:59.285544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:59.285955 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:59.286116 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:59.286125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.287171 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.287177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:59.288237 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:59.288249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.288975 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.288982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:59.289172 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:59.289866 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:59.289885 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:59.292685 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":296,"threads_started":1}
W20260927 05:06:59.293182 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:59.284589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48701 (local address 127.0.97.62:45217)
0927 05:06:59.284852 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:59.284859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:59.285071 (+ 212us) server_negotiation.cc:407] Connection header received
0927 05:06:59.285316 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:59.285327 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:59.285389 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:59.285494 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:59.286269 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.287024 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.288384 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.288813 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.288949 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:59.292914 (+ 3965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48701: BlockingRecv error: recv got EOF from 127.0.97.1:48701 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 27 05:06:59 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:06:59 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:06:59.784453 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:06:59.774968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:56411)
0927 05:06:59.775562 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 05:06:59.775580 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:06:59.775775 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:06:59.776218 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:06:59.776224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:06:59.776651 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:06:59.776820 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:59.776830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.777998 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.778012 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:59.779233 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:06:59.779248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.780305 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.780312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:06:59.780536 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:59.781197 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:06:59.781217 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:06:59.784200 (+ 2983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":326,"threads_started":1}
W20260927 05:06:59.784734 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:06:59.775048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56411 (local address 127.0.97.62:45217)
0927 05:06:59.775379 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 05:06:59.775387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:06:59.775684 (+ 297us) server_negotiation.cc:407] Connection header received
0927 05:06:59.775974 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:06:59.775984 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:06:59.776065 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:06:59.776182 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:06:59.776990 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.777822 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.779402 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:06:59.780138 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:06:59.780287 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:06:59.784441 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56411: BlockingRecv error: recv got EOF from 127.0.97.2:56411 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 27 05:07:00 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:00 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:00.270349 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:00.261395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:47507)
0927 05:07:00.261968 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:00.261985 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:00.262174 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:00.262542 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:00.262547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:00.262949 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:00.263155 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:00.263164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.264358 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.264365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:00.265371 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:00.265384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.266122 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.266128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:00.266336 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:00.266933 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:00.266952 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:00.270013 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":301,"threads_started":1}
W20260927 05:07:00.270498 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:00.261461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47507 (local address 127.0.97.62:45217)
0927 05:07:00.261793 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:00.261801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:00.262086 (+ 285us) server_negotiation.cc:407] Connection header received
0927 05:07:00.262322 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:00.262327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:00.262391 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:00.262505 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:00.263317 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.264213 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.265534 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.265965 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.266111 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:00.270230 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47507: BlockingRecv error: recv got EOF from 127.0.97.3:47507 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Sep 27 05:07:00 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:00 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:00.304427 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:00.294366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:46239)
0927 05:07:00.295077 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:00.295097 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:00.295289 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:00.295701 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:00.295706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:00.296240 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:00.296428 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:00.296440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.297693 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.297700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:00.299014 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:00.299031 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.300005 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.300013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:00.300219 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:00.300890 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:00.300911 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:00.304062 (+ 3151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":372,"threads_started":1}
W20260927 05:07:00.304509 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:00.294490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46239 (local address 127.0.97.62:45217)
0927 05:07:00.294733 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:00.294740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:00.295211 (+ 471us) server_negotiation.cc:407] Connection header received
0927 05:07:00.295440 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:00.295447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:00.295532 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:00.295660 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:00.296624 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.297511 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.299203 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.299779 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.299945 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:00.304290 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46239: BlockingRecv error: recv got EOF from 127.0.97.1:46239 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 05:07:00 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:00 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:00.794713 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:00.785456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:47311)
0927 05:07:00.786135 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:00.786152 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:00.786261 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:00.786688 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:00.786692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:00.787092 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:00.787255 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:00.787264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.788412 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.788416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:00.789373 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:00.789385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.790186 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.790191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:00.790327 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:00.791076 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:00.791098 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:00.794405 (+ 3307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":398,"threads_started":1}
W20260927 05:07:00.795080 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:00.785639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47311 (local address 127.0.97.62:45217)
0927 05:07:00.785820 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:00.785824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:00.786228 (+ 404us) server_negotiation.cc:407] Connection header received
0927 05:07:00.786465 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:00.786469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:00.786531 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:00.786619 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:00.787385 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.788278 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.789543 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:00.790004 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:00.790121 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:00.794723 (+ 4602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47311: BlockingRecv error: recv got EOF from 127.0.97.2:47311 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 05:07:01 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:01 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:01.280411 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:01.271546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:56481)
0927 05:07:01.272134 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:01.272152 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:01.272275 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:01.272655 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:01.272659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:01.273093 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:01.273263 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:01.273272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.274309 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.274312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:01.275240 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:01.275252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.275989 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.275994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:01.276190 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:01.276944 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:01.276963 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:01.280158 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":274,"threads_started":1}
W20260927 05:07:01.280575 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:01.271633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56481 (local address 127.0.97.62:45217)
0927 05:07:01.271879 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:01.271883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:01.272245 (+ 362us) server_negotiation.cc:407] Connection header received
0927 05:07:01.272390 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:01.272394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:01.272459 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:01.272566 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:01.273423 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.274171 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.275390 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.275822 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.275926 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:01.280348 (+ 4422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56481: BlockingRecv error: recv got EOF from 127.0.97.3:56481 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 27 05:07:01 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:01 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:01.313608 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:01.305360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:43697)
0927 05:07:01.305867 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:01.305882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:01.306032 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:01.306358 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:01.306362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:01.306758 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:01.306903 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:01.306912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.307977 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.307981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:01.308864 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:01.308874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.309637 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.309644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:01.309836 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:01.310424 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:01.310440 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:01.313293 (+ 2853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":289,"threads_started":1}
W20260927 05:07:01.313772 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:01.305438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43697 (local address 127.0.97.62:45217)
0927 05:07:01.305639 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:01.305648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:01.305971 (+ 323us) server_negotiation.cc:407] Connection header received
0927 05:07:01.306175 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:01.306179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:01.306235 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:01.306332 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:01.307047 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.307810 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.309009 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.309464 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.309637 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:01.313492 (+ 3855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43697: BlockingRecv error: recv got EOF from 127.0.97.1:43697 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 05:07:01 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:01 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:01.811641 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:01.795901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:57507)
0927 05:07:01.796513 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:01.796529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:01.796714 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:01.797088 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:01.797093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:01.797526 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:01.797688 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:01.797697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.798787 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.798795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:01.799776 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:01.799789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.800729 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.800737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:01.800943 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:01.807592 (+ 6649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:01.807627 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:01.811395 (+ 3768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":345,"threads_started":1}
W20260927 05:07:01.812058 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:01.796019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57507 (local address 127.0.97.62:45217)
0927 05:07:01.796265 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:01.796273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:01.796634 (+ 361us) server_negotiation.cc:407] Connection header received
0927 05:07:01.796842 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:01.796849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:01.796947 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:01.797045 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:01.797856 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.798625 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.800028 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:01.800541 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:01.800702 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:01.811760 (+ 11058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57507: BlockingRecv error: recv got EOF from 127.0.97.2:57507 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 05:07:02 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:02 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:02.290606 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:02.281476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:39319)
0927 05:07:02.282069 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:02.282088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:02.282273 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:02.282660 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:02.282665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:02.283126 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:02.283279 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:02.283288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.284485 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.284492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:02.285466 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:02.285478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.286337 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.286344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:02.286536 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:02.287228 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:02.287246 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:02.290260 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":340,"threads_started":1}
W20260927 05:07:02.290838 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:02.281586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39319 (local address 127.0.97.62:45217)
0927 05:07:02.281846 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:02.281856 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:02.282217 (+ 361us) server_negotiation.cc:407] Connection header received
0927 05:07:02.282407 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:02.282413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:02.282487 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:02.282629 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:02.283443 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.284328 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.285665 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.286147 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.286316 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:02.290508 (+ 4192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39319: BlockingRecv error: recv got EOF from 127.0.97.3:39319 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 05:07:02 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:02 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:02.325302 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:02.314913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:55505)
0927 05:07:02.315667 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:02.315686 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:02.315857 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:02.316379 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:02.316385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:02.317019 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:02.317300 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:02.317317 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.318637 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.318642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:02.319678 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:02.319692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.320509 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.320514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:02.320652 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:02.321387 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:02.321407 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:02.325018 (+ 3611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":349,"threads_started":1}
W20260927 05:07:02.325644 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:02.315010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55505 (local address 127.0.97.62:45217)
0927 05:07:02.315285 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:02.315296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:02.315802 (+ 506us) server_negotiation.cc:407] Connection header received
0927 05:07:02.316052 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:02.316057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:02.316151 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:02.316270 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:02.317590 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.318481 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.319833 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.320354 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.320461 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:02.325360 (+ 4899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55505: BlockingRecv error: recv got EOF from 127.0.97.1:55505 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 05:07:02 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:02 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:02.822095 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:02.812807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:54767)
0927 05:07:02.813435 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:02.813454 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:02.813605 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:02.813957 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:02.813961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:02.814362 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:02.814547 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:02.814556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.815583 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.815587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:02.816655 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:02.816669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.817418 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.817423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:02.817540 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:02.818473 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:02.818500 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:02.821830 (+ 3330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":279,"threads_started":1}
W20260927 05:07:02.822283 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:02.812914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54767 (local address 127.0.97.62:45217)
0927 05:07:02.813255 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:02.813262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:02.813554 (+ 292us) server_negotiation.cc:407] Connection header received
0927 05:07:02.813727 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:02.813731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:02.813799 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:02.813893 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:02.814689 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.815450 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.816814 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:02.817232 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:02.817395 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:02.822037 (+ 4642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54767: BlockingRecv error: recv got EOF from 127.0.97.2:54767 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Sep 27 05:07:03 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:03 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:03.301857 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:03.291913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:41501)
0927 05:07:03.292733 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:03.292753 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:03.292896 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:03.293271 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:03.293275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:03.293690 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:03.293878 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:03.293887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.294989 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.294993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:03.296119 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:03.296133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.297172 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.297179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:03.297371 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:03.298130 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:03.298150 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:03.301575 (+ 3425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":321,"threads_started":1}
W20260927 05:07:03.302110 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:03.292016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41501 (local address 127.0.97.62:45217)
0927 05:07:03.292254 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:03.292259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:03.292861 (+ 602us) server_negotiation.cc:407] Connection header received
0927 05:07:03.293039 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:03.293043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:03.293114 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:03.293216 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:03.294077 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.294852 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.296297 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.296939 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.297125 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:03.301838 (+ 4713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41501: BlockingRecv error: recv got EOF from 127.0.97.3:41501 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 27 05:07:03 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:03 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:03.335547 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:03.326428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:53317)
0927 05:07:03.327043 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:03.327065 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:03.327241 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:03.327592 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:03.327597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:03.328134 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:03.328296 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:03.328305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.329420 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.329426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:03.330421 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:03.330434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.331253 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.331261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:03.331438 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:03.332160 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:03.332179 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:03.335233 (+ 3054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":344,"threads_started":1}
W20260927 05:07:03.335767 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:03.326618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53317 (local address 127.0.97.62:45217)
0927 05:07:03.326908 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:03.326915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:03.327160 (+ 245us) server_negotiation.cc:407] Connection header received
0927 05:07:03.327386 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:03.327390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:03.327452 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:03.327561 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:03.328456 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.329263 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.330587 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.331055 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.331191 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:03.335465 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53317: BlockingRecv error: recv got EOF from 127.0.97.1:53317 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 05:07:03 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:03 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:03.832772 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:03.823214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:57113)
0927 05:07:03.823835 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:03.823858 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:03.824215 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:03.824676 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:03.824685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:03.825242 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:03.825418 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:03.825429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.826619 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.826623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:03.827628 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:03.827641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.828567 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.828572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:03.828704 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:03.829427 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:03.829445 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:03.832516 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":368,"threads_started":1}
W20260927 05:07:03.833146 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:03.823318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57113 (local address 127.0.97.62:45217)
0927 05:07:03.823571 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:03.823580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:03.823982 (+ 402us) server_negotiation.cc:407] Connection header received
0927 05:07:03.824415 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:03.824420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:03.824501 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:03.824597 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:03.825571 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.826463 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.827801 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:03.828372 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:03.828484 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:03.832781 (+ 4297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57113: BlockingRecv error: recv got EOF from 127.0.97.2:57113 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 05:07:04 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:04 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:04.313138 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:04.303185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:43669)
0927 05:07:04.303818 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:04.303842 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:04.304103 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:04.304679 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:04.304683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:04.305162 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:04.305334 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:04.305343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.306587 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.306593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:04.307927 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:04.307943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.308815 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.308820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:04.308975 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:04.309799 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:04.309820 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:04.312879 (+ 3059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":333,"threads_started":1}
W20260927 05:07:04.313503 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:04.303251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43669 (local address 127.0.97.62:45217)
0927 05:07:04.303519 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:04.303529 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:04.303987 (+ 458us) server_negotiation.cc:407] Connection header received
0927 05:07:04.304316 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:04.304322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:04.304420 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:04.304484 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:04.305501 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.306384 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.308135 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.308625 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.308763 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:04.313145 (+ 4382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43669: BlockingRecv error: recv got EOF from 127.0.97.3:43669 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 05:07:04 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:04 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:04.346161 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:04.336688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:46411)
0927 05:07:04.337239 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:04.337256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:04.337422 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:04.337859 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:04.337864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:04.338310 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:04.338477 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:04.338486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.339652 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.339659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:04.340706 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:04.340719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.341655 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.341661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:04.341840 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:04.342533 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:04.342557 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:04.345904 (+ 3347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":270,"threads_started":1}
W20260927 05:07:04.346414 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:04.336763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46411 (local address 127.0.97.62:45217)
0927 05:07:04.337014 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:04.337023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:04.337348 (+ 325us) server_negotiation.cc:407] Connection header received
0927 05:07:04.337571 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:04.337576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:04.337689 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:04.337832 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:04.338632 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.339504 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.340853 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.341488 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.341625 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:04.346127 (+ 4502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46411: BlockingRecv error: recv got EOF from 127.0.97.1:46411 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 05:07:04 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:04 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:04.842557 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:04.833832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:41171)
0927 05:07:04.834425 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:04.834445 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:04.834590 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:04.834979 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:04.834983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:04.835349 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:04.835515 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:04.835523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.836707 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.836711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:04.837681 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:04.837692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.838613 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.838618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:04.838735 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:04.839403 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:04.839420 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:04.842203 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":289,"threads_started":1}
W20260927 05:07:04.842736 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:04.833894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41171 (local address 127.0.97.62:45217)
0927 05:07:04.834146 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:04.834155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:04.834512 (+ 357us) server_negotiation.cc:407] Connection header received
0927 05:07:04.834751 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:04.834758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:04.834852 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:04.834944 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:04.835661 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.836587 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.837852 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:04.838420 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:04.838585 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:04.842479 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41171: BlockingRecv error: recv got EOF from 127.0.97.2:41171 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 05:07:05 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:05 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:05.322584 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:05.314006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:34285)
0927 05:07:05.314546 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:05.314562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:05.314677 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:05.315034 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:05.315037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:05.315403 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:05.315566 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:05.315575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.316609 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.316613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:05.317546 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:05.317557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.318388 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.318393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:05.318518 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:05.319139 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:05.319156 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:05.322180 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
W20260927 05:07:05.322705 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:05.314133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34285 (local address 127.0.97.62:45217)
0927 05:07:05.314308 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:05.314313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:05.314659 (+ 346us) server_negotiation.cc:407] Connection header received
0927 05:07:05.314836 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:05.314840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:05.314903 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:05.314990 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:05.315711 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.316489 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.317776 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.318249 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.318361 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:05.322423 (+ 4062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34285: BlockingRecv error: recv got EOF from 127.0.97.3:34285 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 05:07:05 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:05 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:05.355446 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:05.347090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:53787)
0927 05:07:05.347640 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:05.347660 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:05.347800 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:05.348215 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:05.348220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:05.348750 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:05.348924 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:05.348934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.349946 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.349952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:05.350864 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:05.350875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.351576 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.351581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:05.351762 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:05.352428 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:05.352445 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:05.355229 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":294,"threads_started":1}
W20260927 05:07:05.355681 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:05.347199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53787 (local address 127.0.97.62:45217)
0927 05:07:05.347421 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:05.347430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:05.347728 (+ 298us) server_negotiation.cc:407] Connection header received
0927 05:07:05.347987 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:05.347992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:05.348073 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:05.348191 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:05.349064 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.349807 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.351018 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.351430 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.351564 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:05.355406 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53787: BlockingRecv error: recv got EOF from 127.0.97.1:53787 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 05:07:05 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:05 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:05.851768 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:05.843500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:34643)
0927 05:07:05.844151 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:05.844174 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:05.844290 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:05.844594 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:05.844602 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:05.845085 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:05.845243 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:05.845251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.846323 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.846327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:05.847227 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:05.847237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.848002 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.848007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:05.848144 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:05.848762 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:05.848779 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:05.851470 (+ 2691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":360,"threads_started":1}
W20260927 05:07:05.851893 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:05.843546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34643 (local address 127.0.97.62:45217)
0927 05:07:05.843701 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:05.843705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:05.844351 (+ 646us) server_negotiation.cc:407] Connection header received
0927 05:07:05.844399 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:05.844402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:05.844457 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:05.844531 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:05.845416 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.846199 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.847385 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:05.847838 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:05.848023 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:05.851660 (+ 3637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34643: BlockingRecv error: recv got EOF from 127.0.97.2:34643 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 05:07:06 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:06 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:06.331923 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:06.323536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:38941)
0927 05:07:06.324097 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:06.324118 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:06.324271 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:06.324632 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:06.324636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:06.325103 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:06.325321 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:06.325330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.326376 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.326382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:06.327302 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:06.327312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.328018 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.328032 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:06.328199 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:06.328751 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:06.328768 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:06.331600 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":252,"threads_started":1}
W20260927 05:07:06.332072 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:06.323814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38941 (local address 127.0.97.62:45217)
0927 05:07:06.324049 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:06.324055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:06.324208 (+ 153us) server_negotiation.cc:407] Connection header received
0927 05:07:06.324404 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:06.324410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:06.324475 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:06.324588 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:06.325465 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.326229 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.327455 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.327854 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.328006 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:06.331790 (+ 3784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38941: BlockingRecv error: recv got EOF from 127.0.97.3:38941 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 05:07:06 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:06 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:06.364380 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:06.356335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:35163)
0927 05:07:06.356827 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:06.356843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:06.356955 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:06.357260 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:06.357264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:06.357672 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:06.357841 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:06.357849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.358757 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.358761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:06.359623 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:06.359633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.360472 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.360480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:06.360653 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:06.361348 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:06.361384 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:06.364126 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":267,"threads_started":1}
W20260927 05:07:06.364542 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:06.356413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35163 (local address 127.0.97.62:45217)
0927 05:07:06.356586 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:06.356590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:06.356915 (+ 325us) server_negotiation.cc:407] Connection header received
0927 05:07:06.357048 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:06.357051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:06.357110 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:06.357186 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:06.357964 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.358645 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.359739 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.360315 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.360430 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:06.364323 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35163: BlockingRecv error: recv got EOF from 127.0.97.1:35163 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 05:07:06 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:06 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:06.867086 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:06.852714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:58333)
0927 05:07:06.853265 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:06.853280 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:06.853425 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:06.853770 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:06.853775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:06.854208 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:06.854373 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:06.854381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.855389 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.855395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:06.856777 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:06.856791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.857520 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.857525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:06.857637 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:06.863863 (+ 6226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:06.863880 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:06.866852 (+ 2972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":286,"threads_started":1}
W20260927 05:07:06.867408 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:06.852873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58333 (local address 127.0.97.62:45217)
0927 05:07:06.853107 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:06.853113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:06.853368 (+ 255us) server_negotiation.cc:407] Connection header received
0927 05:07:06.853553 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:06.853558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:06.853626 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:06.853739 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:06.854516 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.855238 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.856916 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:06.857391 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:06.857488 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:06.867120 (+ 9632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58333: BlockingRecv error: recv got EOF from 127.0.97.2:58333 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 05:07:07 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:07 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:07.341176 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:07.332789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:33515)
0927 05:07:07.333260 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:07.333275 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:07.333385 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:07.333685 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:07.333689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:07.334071 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:07.334252 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:07.334260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.335226 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.335229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:07.336183 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:07.336194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.336813 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.336817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:07.336926 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:07.337572 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:07.337590 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:07.340899 (+ 3309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":243,"threads_started":1}
W20260927 05:07:07.341353 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:07.332903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33515 (local address 127.0.97.62:45217)
0927 05:07:07.333151 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:07.333158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:07.333352 (+ 194us) server_negotiation.cc:407] Connection header received
0927 05:07:07.333478 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:07.333481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:07.333539 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:07.333633 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:07.334374 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.335092 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.336297 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.336691 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.336780 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:07.341112 (+ 4332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33515: BlockingRecv error: recv got EOF from 127.0.97.3:33515 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 05:07:07 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:07 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:07.373107 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:07.365289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:51735)
0927 05:07:07.365852 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:07.365868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:07.365976 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:07.366257 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:07.366260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:07.366613 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:07.366785 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:07.366793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.367758 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.367763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:07.368724 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:07.368735 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.369425 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.369428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:07.369542 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:07.370203 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:07.370220 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:07.372890 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":301,"threads_started":1}
W20260927 05:07:07.373265 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:07.365324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51735 (local address 127.0.97.62:45217)
0927 05:07:07.365486 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:07.365490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:07.365937 (+ 447us) server_negotiation.cc:407] Connection header received
0927 05:07:07.366067 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:07.366070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:07.366134 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:07.366213 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:07.366905 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.367630 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.368846 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.369298 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.369403 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:07.373046 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51735: BlockingRecv error: recv got EOF from 127.0.97.1:51735 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 05:07:07 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:07 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:07.876541 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:07.868037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:57729)
0927 05:07:07.868536 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:07.868550 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:07.868721 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:07.869045 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:07.869050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:07.869438 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:07.869580 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:07.869588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.870556 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.870562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:07.871466 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:07.871476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.872233 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.872238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:07.872404 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:07.872977 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:07.872994 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:07.876227 (+ 3233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":261,"threads_started":1}
W20260927 05:07:07.876636 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:07.868089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57729 (local address 127.0.97.62:45217)
0927 05:07:07.868317 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:07.868325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:07.868637 (+ 312us) server_negotiation.cc:407] Connection header received
0927 05:07:07.868852 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:07.868857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:07.868926 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:07.869021 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:07.869713 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.870423 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.871612 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:07.872088 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:07.872225 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:07.876400 (+ 4175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57729: BlockingRecv error: recv got EOF from 127.0.97.2:57729 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 05:07:08 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:08 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:08.349931 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:08.342030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:52353)
0927 05:07:08.342548 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:08.342563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:08.342676 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:08.342933 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:08.342936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:08.343348 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:08.343510 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:08.343519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.344581 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.344584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:08.345446 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:08.345457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.346092 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.346095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:08.346214 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:08.346791 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:08.346807 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:08.349708 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":285,"threads_started":1}
W20260927 05:07:08.350143 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:08.342076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52353 (local address 127.0.97.62:45217)
0927 05:07:08.342253 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:08.342257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:08.342632 (+ 375us) server_negotiation.cc:407] Connection header received
0927 05:07:08.342752 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:08.342756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:08.342810 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:08.342886 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:08.343616 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.344413 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.345560 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.345977 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.346065 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:08.349893 (+ 3828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52353: BlockingRecv error: recv got EOF from 127.0.97.3:52353 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 05:07:08 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:08 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:08.381568 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:08.374045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:33647)
0927 05:07:08.374565 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:08.374581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:08.374688 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:08.374957 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:08.374959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:08.375317 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:08.375471 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:08.375479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.376438 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.376441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:08.377329 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:08.377339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.377987 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.377990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:08.378101 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:08.378734 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:08.378752 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:08.381351 (+ 2599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":307,"threads_started":1}
W20260927 05:07:08.381716 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:08.374143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33647 (local address 127.0.97.62:45217)
0927 05:07:08.374375 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:08.374380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:08.374660 (+ 280us) server_negotiation.cc:407] Connection header received
0927 05:07:08.374783 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:08.374786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:08.374843 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:08.374918 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:08.375587 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.376331 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.377446 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.377830 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.377936 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:08.381513 (+ 3577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33647: BlockingRecv error: recv got EOF from 127.0.97.1:33647 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 05:07:08 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:08 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:08.887228 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:08.877601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:36907)
0927 05:07:08.878120 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:08.878143 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:08.878326 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:08.878796 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:08.878803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:08.879329 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:08.879526 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:08.879539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.880682 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.880688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:08.882173 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:08.882192 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.882932 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.882937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:08.883096 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:08.883756 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:08.883784 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:08.886974 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":259,"threads_started":1}
W20260927 05:07:08.887573 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:08.877715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36907 (local address 127.0.97.62:45217)
0927 05:07:08.877868 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:08.877872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:08.878212 (+ 340us) server_negotiation.cc:407] Connection header received
0927 05:07:08.878453 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:08.878460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:08.878535 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:08.878630 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:08.879706 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.880533 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.882319 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:08.882794 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:08.882922 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:08.887268 (+ 4346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36907: BlockingRecv error: recv got EOF from 127.0.97.2:36907 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 05:07:09 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:09 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:09.359076 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:09.351138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:38669)
0927 05:07:09.351701 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:09.351717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:09.351864 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:09.352282 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:09.352287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:09.352711 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:09.352856 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:09.352865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.353831 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.353837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:09.354856 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:09.354867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.355564 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.355570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:09.355748 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:09.356279 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:09.356295 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:09.358855 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":270,"threads_started":1}
W20260927 05:07:09.359275 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:09.351285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38669 (local address 127.0.97.62:45217)
0927 05:07:09.351532 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:09.351538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:09.351793 (+ 255us) server_negotiation.cc:407] Connection header received
0927 05:07:09.352017 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:09.352031 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:09.352110 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:09.352249 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:09.353008 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.353702 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.354996 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.355398 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.355528 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:09.359032 (+ 3504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38669: BlockingRecv error: recv got EOF from 127.0.97.3:38669 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 05:07:09 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:09 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:09.390848 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:09.382652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:50349)
0927 05:07:09.383111 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:09.383146 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:09.383260 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:09.383539 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:09.383541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:09.383884 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:09.384129 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:09.384139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.385326 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.385331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:09.386525 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:09.386538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.387164 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.387167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:09.387276 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:09.387904 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:09.387921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:09.390601 (+ 2680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":248,"threads_started":1}
W20260927 05:07:09.391196 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:09.382695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50349 (local address 127.0.97.62:45217)
0927 05:07:09.382910 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:09.382914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:09.383228 (+ 314us) server_negotiation.cc:407] Connection header received
0927 05:07:09.383354 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:09.383357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:09.383416 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:09.383497 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:09.384253 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.385191 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.386645 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.387051 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.387139 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:09.390828 (+ 3689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50349: BlockingRecv error: recv got EOF from 127.0.97.1:50349 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 27 05:07:09 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:09 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:09.895979 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:09.888116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:32789)
0927 05:07:09.888605 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:09.888620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:09.888731 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:09.889007 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:09.889010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:09.889344 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:09.889504 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:09.889512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.890571 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.890574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:09.891599 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:09.891610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.892252 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.892256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:09.892368 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:09.892912 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:09.892928 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:09.895631 (+ 2703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":256,"threads_started":1}
W20260927 05:07:09.896024 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:09.888294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32789 (local address 127.0.97.62:45217)
0927 05:07:09.888454 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:09.888458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:09.888692 (+ 234us) server_negotiation.cc:407] Connection header received
0927 05:07:09.888823 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:09.888826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:09.888884 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:09.888964 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:09.889662 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.890454 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.891721 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:09.892139 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:09.892227 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:09.895806 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:32789: BlockingRecv error: recv got EOF from 127.0.97.2:32789 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 05:07:10 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:10 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:10.367470 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:10.359989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.3:51555)
0927 05:07:10.360474 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:10.360491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:10.360628 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:10.360908 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:10.360911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:10.361256 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:10.361418 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:10.361426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.362356 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.362360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:10.363232 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:10.363243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.363822 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.363826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:10.363987 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:10.364550 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:10.364565 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:10.367231 (+ 2666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":282,"threads_started":1}
W20260927 05:07:10.367686 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:10.360234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51555 (local address 127.0.97.62:45217)
0927 05:07:10.360439 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:10.360446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:10.360574 (+ 128us) server_negotiation.cc:407] Connection header received
0927 05:07:10.360725 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:10.360728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:10.360786 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:10.360862 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:10.361533 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.362240 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.363345 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.363710 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.363801 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:10.367422 (+ 3621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51555: BlockingRecv error: recv got EOF from 127.0.97.3:51555 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 05:07:10 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:10 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:10.400554 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:10.392012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.1:38449)
0927 05:07:10.392644 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:10.392660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:10.392794 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:10.393162 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:10.393166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:10.393622 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:10.393805 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:10.393813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.394880 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.394884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:10.395778 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:10.395788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.396487 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.396493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:10.396662 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:10.397206 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:10.397223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:10.400198 (+ 2975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":347,"threads_started":1}
W20260927 05:07:10.400607 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:10.392180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38449 (local address 127.0.97.62:45217)
0927 05:07:10.392376 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:10.392384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:10.392741 (+ 357us) server_negotiation.cc:407] Connection header received
0927 05:07:10.392922 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:10.392927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:10.393033 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:10.393125 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:10.393978 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.394755 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.395892 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.396339 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.396474 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:10.400379 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38449: BlockingRecv error: recv got EOF from 127.0.97.1:38449 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 05:07:10 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 05:07:10 dist-test-slave-hgs0 krb5kdc[8653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 05:07:10.905193 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:10.897175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45217 (local address 127.0.97.2:50209)
0927 05:07:10.897651 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:10.897668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:10.897778 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:10.898053 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:10.898056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:10.898442 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:10.898612 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:10.898620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.899707 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.899711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:10.900813 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:10.900824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.901534 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.901539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:10.901700 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:10.902277 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:10.902294 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:10.904949 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:45217: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":265,"threads_started":1}
W20260927 05:07:10.905339 9391 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:10.897245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50209 (local address 127.0.97.62:45217)
0927 05:07:10.897452 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:10.897461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:10.897745 (+ 284us) server_negotiation.cc:407] Connection header received
0927 05:07:10.897882 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:10.897885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:10.897943 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:10.898020 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:10.898731 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.899580 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.900949 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:10.901374 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:10.901518 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:10.905120 (+ 3602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50209: BlockingRecv error: recv got EOF from 127.0.97.2:50209 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:07:11.203852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9242
I20260927 05:07:11.216127 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:11.205054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.0.97.1:33821)
0927 05:07:11.205588 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:11.205596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:11.205623 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:07:11.205687 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:11.205693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:11.205935 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:11.206096 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:11.207483 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:11.208509 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:11.209664 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:11.210218 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:11.210318 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:11.212792 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:11.212801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:11.212815 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:11.212882 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:11.215463 (+ 2581us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 05:07:11.215783 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38248: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":282,"threads_started":1}
W20260927 05:07:11.216188 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9242: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33821: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 05:07:11.216246 9702 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38248: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 05:07:11.216392 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 9242
I20260927 05:07:11.230106 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9393
I20260927 05:07:11.242497 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:11.231162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.0.97.2:43111)
0927 05:07:11.231796 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:11.231804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:11.231833 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:07:11.231891 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:11.231897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:11.232233 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:11.232384 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:11.233574 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:11.234726 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:11.236145 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:11.236692 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:11.236790 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:11.239224 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:11.239232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:11.239244 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:11.239282 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:11.241900 (+ 2618us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 05:07:11.242221 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32990: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":336,"threads_started":1}
W20260927 05:07:11.242604 9703 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32990: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 05:07:11.242691 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9393: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:43111: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 05:07:11.242841 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 9393
I20260927 05:07:11.259617 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9524
I20260927 05:07:11.270902 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:11.260544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.0.97.3:39241)
0927 05:07:11.261113 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:11.261121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:11.261147 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:07:11.261211 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:11.261217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:11.261444 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:11.261596 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:11.262829 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:11.263833 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:11.265059 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:11.265580 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:11.265679 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:11.268115 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:11.268123 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:11.268136 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:11.268178 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:11.270442 (+ 2264us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 05:07:11.270677 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41640: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":300,"threads_started":1}
W20260927 05:07:11.271018 9704 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41640: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 05:07:11.271005 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9524: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:39241: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 05:07:11.271169 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 9524
I20260927 05:07:11.286787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9173
I20260927 05:07:11.288038 9235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:11.400955 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 9173
2026-09-27T05:07:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21603 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T05:07:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:07:11Z Disabled control of system clock
I20260927 05:07:11.475142 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45047
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 05:07:11.791296 9709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:11.791567 9709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:11.791643 9709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:11.798748 9709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:07:11.798843 9709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:11.798879 9709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:11.798928 9709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:07:11.798964 9709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:07:11.807874 9709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:11.809080 9709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:11.810739 9709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:11.822582 9714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:11.822520 9715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:11.822554 9717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:11.823056 9709 server_base.cc:1034] running on GCE node
I20260927 05:07:11.823859 9709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:11.825237 9709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:11.826440 9709 hybrid_clock.cc:648] HybridClock initialized: now 1790485631826398 us; error 24 us; skew 500 ppm
I20260927 05:07:11.826781 9709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:07:11.829020 9709 webserver.cc:460] Webserver started at http://127.0.97.62:37211/ using document root <none> and password file <none>
I20260927 05:07:11.829710 9709 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:11.829823 9709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:11.830163 9709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:11.833107 9709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "e49fbabe50c94af2aa4b26fc1bf05db0"
format_stamp: "Formatted at 2026-09-27 05:07:11 on dist-test-slave-hgs0"
server_key: "488cc3bd409598c1fdf0b93f29de2074"
server_key_iv: "292fbc47eb4eca04b5727f5e1b3c4b64"
server_key_version: "encryptionkey@0"
I20260927 05:07:11.833875 9709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "e49fbabe50c94af2aa4b26fc1bf05db0"
format_stamp: "Formatted at 2026-09-27 05:07:11 on dist-test-slave-hgs0"
server_key: "488cc3bd409598c1fdf0b93f29de2074"
server_key_iv: "292fbc47eb4eca04b5727f5e1b3c4b64"
server_key_version: "encryptionkey@0"
I20260927 05:07:11.838711 9709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 05:07:11.842013 9723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:11.843551 9709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:07:11.843765 9709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "e49fbabe50c94af2aa4b26fc1bf05db0"
format_stamp: "Formatted at 2026-09-27 05:07:11 on dist-test-slave-hgs0"
server_key: "488cc3bd409598c1fdf0b93f29de2074"
server_key_iv: "292fbc47eb4eca04b5727f5e1b3c4b64"
server_key_version: "encryptionkey@0"
I20260927 05:07:11.844002 9709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:11.860208 9709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:11.860967 9709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:11.861248 9709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:11.879151 9709 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45047
I20260927 05:07:11.879172 9774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45047 every 8 connection(s)
I20260927 05:07:11.880466 9709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:07:11.883437 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 9709
I20260927 05:07:11.883651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:07:11.884155 9775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:07:11.884393 388 external_mini_cluster.cc:1428] Setting key 62a6e9976abfb2ebd7da931503f40a5e
I20260927 05:07:11.898005 9775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0: Bootstrap starting.
I20260927 05:07:11.901366 9775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0: Neither blocks nor log segments found. Creating new log.
I20260927 05:07:11.902573 9775 log.cc:824] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0: Log is configured to *not* fsync() on all Append() calls
I20260927 05:07:11.903053 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:11.886310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.0.97.62:45047)
0927 05:07:11.887360 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:11.887379 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:11.887429 (+ 50us) server_negotiation.cc:407] Connection header received
0927 05:07:11.889141 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:11.889172 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:11.889757 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:11.890176 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:11.890707 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:11.892693 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:11.893986 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:11.894996 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:11.895170 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:11.900298 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:11.900309 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:11.900326 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:11.901005 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:11.901118 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:11.901150 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:11.901278 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:11.901506 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 05:07:11.901874 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":332,"threads_started":1}
I20260927 05:07:11.905474 9775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0: No bootstrap required, opened a new log
I20260927 05:07:11.912506 9775 raft_consensus.cc:348] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49fbabe50c94af2aa4b26fc1bf05db0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45047 } }
I20260927 05:07:11.912834 9775 raft_consensus.cc:374] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:07:11.912967 9775 raft_consensus.cc:729] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49fbabe50c94af2aa4b26fc1bf05db0, State: Initialized, Role: FOLLOWER
I20260927 05:07:11.913561 9775 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49fbabe50c94af2aa4b26fc1bf05db0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45047 } }
I20260927 05:07:11.913780 9775 raft_consensus.cc:388] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:07:11.913899 9775 raft_consensus.cc:482] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:07:11.914064 9775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:07:11.916107 9775 raft_consensus.cc:504] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49fbabe50c94af2aa4b26fc1bf05db0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45047 } }
I20260927 05:07:11.916580 9775 leader_election.cc:302] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e49fbabe50c94af2aa4b26fc1bf05db0; no voters:
I20260927 05:07:11.917168 9775 leader_election.cc:288] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:07:11.917338 9780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:07:11.918530 9780 raft_consensus.cc:686] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [term 1 LEADER]: Becoming Leader. State: Replica: e49fbabe50c94af2aa4b26fc1bf05db0, State: Running, Role: LEADER
I20260927 05:07:11.919008 9780 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49fbabe50c94af2aa4b26fc1bf05db0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45047 } }
I20260927 05:07:11.919272 9775 sys_catalog.cc:564] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:07:11.921885 9782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49fbabe50c94af2aa4b26fc1bf05db0. Latest consensus state: current_term: 1 leader_uuid: "e49fbabe50c94af2aa4b26fc1bf05db0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49fbabe50c94af2aa4b26fc1bf05db0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45047 } } }
I20260927 05:07:11.923002 9782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [sys.catalog]: This master's current role is: LEADER
I20260927 05:07:11.922102 9781 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e49fbabe50c94af2aa4b26fc1bf05db0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49fbabe50c94af2aa4b26fc1bf05db0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45047 } } }
I20260927 05:07:11.923480 9781 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0 [sys.catalog]: This master's current role is: LEADER
I20260927 05:07:11.928915 9790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:07:11.934000 9790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:07:11.941869 9790 catalog_manager.cc:1347] Generated new cluster ID: a7e01c54badb480db3af81882799e5b1
I20260927 05:07:11.941983 9790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:07:11.978603 9790 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:07:11.980468 9790 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:07:11.990594 9790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e49fbabe50c94af2aa4b26fc1bf05db0: Generated new TSK 0
I20260927 05:07:11.991663 9790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:07:12.001092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45047
--builtin_ntp_servers=127.0.97.20:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 05:07:12.324182 9799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:12.324476 9799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:12.324627 9799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:12.331558 9799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:12.331684 9799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:12.331866 9799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:07:12.340834 9799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:12.342072 9799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:12.344274 9799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:12.356482 9804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:12.357249 9805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:12.358362 9807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:12.358883 9799 server_base.cc:1034] running on GCE node
I20260927 05:07:12.359457 9799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:12.360569 9799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:12.361817 9799 hybrid_clock.cc:648] HybridClock initialized: now 1790485632361757 us; error 44 us; skew 500 ppm
I20260927 05:07:12.362138 9799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:07:12.364710 9799 webserver.cc:460] Webserver started at http://127.0.97.1:43963/ using document root <none> and password file <none>
I20260927 05:07:12.365377 9799 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:12.365468 9799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:12.365759 9799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:12.368717 9799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "ec6473fcd2444745962bfeed3f84b75f"
format_stamp: "Formatted at 2026-09-27 05:07:12 on dist-test-slave-hgs0"
server_key: "77f5e77099900b9aca64c52158670049"
server_key_iv: "00be5aa7f96ba112ca0a92218db31190"
server_key_version: "encryptionkey@0"
I20260927 05:07:12.369483 9799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ec6473fcd2444745962bfeed3f84b75f"
format_stamp: "Formatted at 2026-09-27 05:07:12 on dist-test-slave-hgs0"
server_key: "77f5e77099900b9aca64c52158670049"
server_key_iv: "00be5aa7f96ba112ca0a92218db31190"
server_key_version: "encryptionkey@0"
I20260927 05:07:12.374632 9799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 05:07:12.377864 9813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:12.379118 9799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 05:07:12.379256 9799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "ec6473fcd2444745962bfeed3f84b75f"
format_stamp: "Formatted at 2026-09-27 05:07:12 on dist-test-slave-hgs0"
server_key: "77f5e77099900b9aca64c52158670049"
server_key_iv: "00be5aa7f96ba112ca0a92218db31190"
server_key_version: "encryptionkey@0"
I20260927 05:07:12.379422 9799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:12.401268 9799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:12.402004 9799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:12.402313 9799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:12.403200 9799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:07:12.404778 9799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:07:12.404894 9799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:12.405023 9799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:07:12.405108 9799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:12.437709 9799 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42311
I20260927 05:07:12.437793 9925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42311 every 8 connection(s)
I20260927 05:07:12.438961 9799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:07:12.445051 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 9799
I20260927 05:07:12.445253 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:07:12.446103 388 external_mini_cluster.cc:1428] Setting key 5ddfcd5ab3ba21b0e04eef0b724d2a63
I20260927 05:07:12.449105 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45047
--builtin_ntp_servers=127.0.97.20:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 05:07:12.452934 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:12.441675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36623 (local address 127.0.97.62:45047)
0927 05:07:12.442090 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:12.442095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:12.443331 (+ 1236us) server_negotiation.cc:407] Connection header received
0927 05:07:12.445207 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:12.445212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:12.445305 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:12.445430 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:12.446878 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:12.447799 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:12.450226 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:12.450599 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:12.450699 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:12.452256 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:12.452261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:12.452264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:12.452280 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:12.452316 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:12.452327 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:12.452396 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:12.452639 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 05:07:12.452707 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":221,"threads_started":1}
I20260927 05:07:12.454360 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:12.442306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45047 (local address 127.0.97.1:36623)
0927 05:07:12.443182 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:12.443217 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:12.444932 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:12.445711 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:12.445724 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:12.445864 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:12.446687 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:12.446714 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:12.448013 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:12.448017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:12.450072 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:12.450091 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:12.450814 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:12.450823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:12.451169 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:12.451893 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:07:12.451898 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:12.451967 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:07:12.452071 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:07:12.452423 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:07:12.452436 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:07:12.452445 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 05:07:12.452609 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 05:07:12.453008 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":257,"threads_started":1}
I20260927 05:07:12.455986 9926 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45047
I20260927 05:07:12.456305 9926 heartbeater.cc:464] Registering TS with master...
I20260927 05:07:12.457069 9926 heartbeater.cc:511] Master 127.0.97.62:45047 requested a full tablet report, sending...
I20260927 05:07:12.459030 9740 ts_manager.cc:194] Registered new tserver with Master: ec6473fcd2444745962bfeed3f84b75f (127.0.97.1:42311)
I20260927 05:07:12.460623 9740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:36623
W20260927 05:07:12.765508 9930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:12.765857 9930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:12.766027 9930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:12.773017 9930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:12.773146 9930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:12.773329 9930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:07:12.782063 9930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:12.783399 9930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:12.785244 9930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:12.798202 9936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:12.798259 9935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:12.798633 9938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:12.798959 9930 server_base.cc:1034] running on GCE node
I20260927 05:07:12.799530 9930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:12.800547 9930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:12.801743 9930 hybrid_clock.cc:648] HybridClock initialized: now 1790485632801680 us; error 54 us; skew 500 ppm
I20260927 05:07:12.802042 9930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:07:12.804556 9930 webserver.cc:460] Webserver started at http://127.0.97.2:46865/ using document root <none> and password file <none>
I20260927 05:07:12.805229 9930 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:12.805343 9930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:12.805665 9930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:12.808508 9930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "d0492251364249889fd029b7a450b590"
format_stamp: "Formatted at 2026-09-27 05:07:12 on dist-test-slave-hgs0"
server_key: "d59888c8ab5608aec9386789fd56121e"
server_key_iv: "1b4b8e2e6dfe2b05bce0e6640a23c34b"
server_key_version: "encryptionkey@0"
I20260927 05:07:12.809291 9930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d0492251364249889fd029b7a450b590"
format_stamp: "Formatted at 2026-09-27 05:07:12 on dist-test-slave-hgs0"
server_key: "d59888c8ab5608aec9386789fd56121e"
server_key_iv: "1b4b8e2e6dfe2b05bce0e6640a23c34b"
server_key_version: "encryptionkey@0"
I20260927 05:07:12.814592 9930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 05:07:12.817785 9944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:12.819098 9930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:07:12.819281 9930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "d0492251364249889fd029b7a450b590"
format_stamp: "Formatted at 2026-09-27 05:07:12 on dist-test-slave-hgs0"
server_key: "d59888c8ab5608aec9386789fd56121e"
server_key_iv: "1b4b8e2e6dfe2b05bce0e6640a23c34b"
server_key_version: "encryptionkey@0"
I20260927 05:07:12.819490 9930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:12.832767 9930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:12.833534 9930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:12.833822 9930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:12.834707 9930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:07:12.836153 9930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:07:12.836267 9930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:12.836397 9930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:07:12.836480 9930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:12.872642 9930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34075
I20260927 05:07:12.872704 10056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34075 every 8 connection(s)
I20260927 05:07:12.874307 9930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:07:12.881613 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 9930
I20260927 05:07:12.881796 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:07:12.882210 388 external_mini_cluster.cc:1428] Setting key ffb2a2e2817c2284e3124da3d77c3834
I20260927 05:07:12.885210 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45047
--builtin_ntp_servers=127.0.97.20:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 05:07:12.887684 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:12.876839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57775 (local address 127.0.97.62:45047)
0927 05:07:12.877014 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:12.877018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:12.878422 (+ 1404us) server_negotiation.cc:407] Connection header received
0927 05:07:12.880302 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:12.880308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:12.880398 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:12.880501 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:12.881625 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:12.882758 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:12.884906 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:12.885349 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:12.885471 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:12.886887 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:12.886894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:12.886898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:12.886925 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:12.886953 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:12.886964 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:12.887057 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:12.887274 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 05:07:12.887366 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:07:12.888777 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:12.877346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45047 (local address 127.0.97.2:57775)
0927 05:07:12.878230 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:12.878280 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:12.879919 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:12.880703 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:12.880711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:12.880792 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:12.881446 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:12.881471 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:12.882925 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:12.882934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:12.884767 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:12.884778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:12.885692 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:12.885697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:12.885903 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:12.886587 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:07:12.886590 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:12.886627 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:07:12.886706 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:07:12.887057 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:07:12.887068 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:07:12.887078 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:07:12.887266 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 05:07:12.887681 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":270,"threads_started":1}
I20260927 05:07:12.890141 10057 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45047
I20260927 05:07:12.890444 10057 heartbeater.cc:464] Registering TS with master...
I20260927 05:07:12.891158 10057 heartbeater.cc:511] Master 127.0.97.62:45047 requested a full tablet report, sending...
I20260927 05:07:12.892513 9740 ts_manager.cc:194] Registered new tserver with Master: d0492251364249889fd029b7a450b590 (127.0.97.2:34075)
I20260927 05:07:12.893646 9740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:57775
W20260927 05:07:13.211742 10060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:13.212121 10060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:13.212289 10060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:13.219439 10060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:13.219573 10060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:13.219777 10060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:07:13.229010 10060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:13.230326 10060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:13.232033 10060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:13.244769 10066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:13.244761 10065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:13.244834 10068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:13.245630 10060 server_base.cc:1034] running on GCE node
I20260927 05:07:13.246248 10060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:13.247357 10060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:13.248572 10060 hybrid_clock.cc:648] HybridClock initialized: now 1790485633248514 us; error 49 us; skew 500 ppm
I20260927 05:07:13.248879 10060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:07:13.251067 10060 webserver.cc:460] Webserver started at http://127.0.97.3:40209/ using document root <none> and password file <none>
I20260927 05:07:13.251742 10060 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:13.251873 10060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:13.252246 10060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:13.254956 10060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "3ded37a293344325a71ed0e6d13711f0"
format_stamp: "Formatted at 2026-09-27 05:07:13 on dist-test-slave-hgs0"
server_key: "0b06612ad482bb57af2f9134ffd64670"
server_key_iv: "048d75c53f95cd5341264581c77334fb"
server_key_version: "encryptionkey@0"
I20260927 05:07:13.255731 10060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3ded37a293344325a71ed0e6d13711f0"
format_stamp: "Formatted at 2026-09-27 05:07:13 on dist-test-slave-hgs0"
server_key: "0b06612ad482bb57af2f9134ffd64670"
server_key_iv: "048d75c53f95cd5341264581c77334fb"
server_key_version: "encryptionkey@0"
I20260927 05:07:13.260736 10060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:07:13.264190 10074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:13.265466 10060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:07:13.265640 10060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "3ded37a293344325a71ed0e6d13711f0"
format_stamp: "Formatted at 2026-09-27 05:07:13 on dist-test-slave-hgs0"
server_key: "0b06612ad482bb57af2f9134ffd64670"
server_key_iv: "048d75c53f95cd5341264581c77334fb"
server_key_version: "encryptionkey@0"
I20260927 05:07:13.265868 10060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:13.280680 10060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:13.281512 10060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:13.281790 10060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:13.282724 10060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:07:13.284204 10060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:07:13.284319 10060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:13.284456 10060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:07:13.284541 10060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:13.316478 10060 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34327
I20260927 05:07:13.316527 10186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34327 every 8 connection(s)
I20260927 05:07:13.317746 10060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:07:13.321915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 10060
I20260927 05:07:13.322170 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:07:13.322744 388 external_mini_cluster.cc:1428] Setting key 212c4b00fea8917d8505bb1ed5fc6c5a
I20260927 05:07:13.331069 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:13.320321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56027 (local address 127.0.97.62:45047)
0927 05:07:13.320513 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:13.320520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:13.322278 (+ 1758us) server_negotiation.cc:407] Connection header received
0927 05:07:13.324600 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:13.324603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:13.324723 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:13.324826 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:13.326528 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:13.327232 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:13.328755 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:13.329155 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:13.329253 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:13.330424 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:13.330429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:13.330432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:13.330449 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:13.330485 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:13.330497 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:13.330589 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:13.330766 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 05:07:13.330840 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:07:13.332050 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:13.320904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45047 (local address 127.0.97.3:56027)
0927 05:07:13.322085 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:13.322137 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:13.324299 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:13.325707 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:13.325725 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:13.325826 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:13.326387 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:13.326402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:13.327349 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:13.327352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:13.328620 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:13.328631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:13.329329 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:13.329333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:13.329517 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:13.330128 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:07:13.330131 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:13.330166 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:07:13.330243 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:07:13.330588 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:07:13.330594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:07:13.330599 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 05:07:13.330731 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 05:07:13.330998 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":372,"threads_started":1}
I20260927 05:07:13.333387 10187 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45047
I20260927 05:07:13.333628 10187 heartbeater.cc:464] Registering TS with master...
I20260927 05:07:13.334290 10187 heartbeater.cc:511] Master 127.0.97.62:45047 requested a full tablet report, sending...
I20260927 05:07:13.335479 9740 ts_manager.cc:194] Registered new tserver with Master: 3ded37a293344325a71ed0e6d13711f0 (127.0.97.3:34327)
I20260927 05:07:13.336376 9740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:56027
I20260927 05:07:13.337864 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 05:07:13.343483 9928 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:13.342032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.0.97.62:45047)
0927 05:07:13.342215 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:13.342219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:13.342665 (+ 446us) server_negotiation.cc:407] Connection header received
0927 05:07:13.342789 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:13.342792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:13.342854 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:13.342931 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:13.343259 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37642: BlockingRecv error: recv got EOF from 127.0.0.1:37642 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:07:13.344331 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9799
I20260927 05:07:13.351481 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:13.345178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.0.97.1:42311)
0927 05:07:13.345694 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:13.345700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:13.345724 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:07:13.345773 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:13.345779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:13.345975 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:13.346092 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:13.346486 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:13.347422 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:13.348774 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:13.349291 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:13.349387 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:13.350269 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:13.350276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:13.350289 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:13.350835 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:13.350878 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:13.350896 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:13.350979 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:13.351115 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 05:07:13.351216 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":262,"threads_started":1}
I20260927 05:07:13.352639 9921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:13.432951 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 9799
I20260927 05:07:13.449486 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9930
I20260927 05:07:13.456704 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:13.450391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.0.97.2:34075)
0927 05:07:13.450909 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:13.450923 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:13.450952 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:07:13.451011 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:13.451018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:13.451250 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:13.451376 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:13.451704 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:13.452727 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:13.453836 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:13.454373 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:13.454467 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:13.455361 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:13.455368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:13.455381 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:13.456033 (+ 652us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:13.456092 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:13.456120 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:13.456215 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:13.456309 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 05:07:13.456422 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":284,"threads_started":1}
I20260927 05:07:13.457880 10052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:13.537657 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 9930
I20260927 05:07:13.554189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 10060
I20260927 05:07:13.562181 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:13.555110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.0.97.3:34327)
0927 05:07:13.555674 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:13.555680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:13.555710 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:07:13.555774 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:13.555780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:13.556052 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:13.556219 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:13.556643 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:13.557684 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:13.558991 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:13.559529 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:13.559687 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:13.560775 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:13.560783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:13.560796 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:13.561411 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:13.561462 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:13.561480 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:13.561603 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:13.561719 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 05:07:13.561852 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":292,"threads_started":1}
I20260927 05:07:13.563388 10182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:13.644512 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 10060
I20260927 05:07:13.660386 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 9709
I20260927 05:07:13.661561 9770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:13.760447 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 9709
2026-09-27T05:07:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2354 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T05:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:07:13Z Disabled control of system clock
I20260927 05:07:13.829599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44967
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 05:07:14.147233 10206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:14.147541 10206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:14.147635 10206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:14.154917 10206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:07:14.155032 10206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:14.155083 10206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:14.155151 10206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:07:14.155201 10206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:07:14.165025 10206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:14.166306 10206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:14.168150 10206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:14.181254 10211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:14.181358 10212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:14.181456 10214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:14.183037 10206 server_base.cc:1034] running on GCE node
I20260927 05:07:14.183810 10206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:14.185360 10206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:14.186580 10206 hybrid_clock.cc:648] HybridClock initialized: now 1790485634186505 us; error 51 us; skew 500 ppm
I20260927 05:07:14.186909 10206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:07:14.189196 10206 webserver.cc:460] Webserver started at http://127.0.97.62:40087/ using document root <none> and password file <none>
I20260927 05:07:14.189831 10206 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:14.189914 10206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:14.190169 10206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:14.193032 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "0abfc0c3f9ca494697dad0b4ab656547"
format_stamp: "Formatted at 2026-09-27 05:07:14 on dist-test-slave-hgs0"
server_key: "1b513157fba216d614b9bdda7c24fb68"
server_key_iv: "7660a0fa60c2ccf4f86f263a49769dfd"
server_key_version: "encryptionkey@0"
I20260927 05:07:14.193800 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "0abfc0c3f9ca494697dad0b4ab656547"
format_stamp: "Formatted at 2026-09-27 05:07:14 on dist-test-slave-hgs0"
server_key: "1b513157fba216d614b9bdda7c24fb68"
server_key_iv: "7660a0fa60c2ccf4f86f263a49769dfd"
server_key_version: "encryptionkey@0"
I20260927 05:07:14.198693 10206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:07:14.201898 10220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:14.203155 10206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:07:14.203290 10206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "0abfc0c3f9ca494697dad0b4ab656547"
format_stamp: "Formatted at 2026-09-27 05:07:14 on dist-test-slave-hgs0"
server_key: "1b513157fba216d614b9bdda7c24fb68"
server_key_iv: "7660a0fa60c2ccf4f86f263a49769dfd"
server_key_version: "encryptionkey@0"
I20260927 05:07:14.203459 10206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:14.208945 10206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:14.209663 10206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:14.209913 10206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:14.227669 10206 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44967
I20260927 05:07:14.227680 10271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44967 every 8 connection(s)
I20260927 05:07:14.229094 10206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:07:14.232678 10272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:07:14.238739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 10206
I20260927 05:07:14.238942 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:07:14.239562 388 external_mini_cluster.cc:1428] Setting key 317b1b7dd1883cfc3e9397f0560ed142
I20260927 05:07:14.244498 10272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547: Bootstrap starting.
I20260927 05:07:14.248565 10272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547: Neither blocks nor log segments found. Creating new log.
I20260927 05:07:14.249409 10275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:14.241757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.0.97.62:44967)
0927 05:07:14.242811 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:14.242825 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:14.242876 (+ 51us) server_negotiation.cc:407] Connection header received
0927 05:07:14.245217 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:14.245247 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:14.245746 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:14.246235 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:14.247332 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:14.247340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:14.247357 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:14.247377 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:14.247445 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:14.247470 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:14.247685 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:14.247811 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 05:07:14.248106 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":317,"threads_started":1}
I20260927 05:07:14.249878 10272 log.cc:824] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547: Log is configured to *not* fsync() on all Append() calls
I20260927 05:07:14.253182 10272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547: No bootstrap required, opened a new log
I20260927 05:07:14.260634 10272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc0c3f9ca494697dad0b4ab656547" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44967 } }
I20260927 05:07:14.261101 10272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:07:14.261212 10272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0abfc0c3f9ca494697dad0b4ab656547, State: Initialized, Role: FOLLOWER
I20260927 05:07:14.261763 10272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc0c3f9ca494697dad0b4ab656547" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44967 } }
I20260927 05:07:14.261988 10272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:07:14.262127 10272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:07:14.262302 10272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:07:14.264650 10272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc0c3f9ca494697dad0b4ab656547" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44967 } }
I20260927 05:07:14.265244 10272 leader_election.cc:302] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0abfc0c3f9ca494697dad0b4ab656547; no voters:
I20260927 05:07:14.265851 10272 leader_election.cc:288] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:07:14.265964 10277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:07:14.266774 10277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [term 1 LEADER]: Becoming Leader. State: Replica: 0abfc0c3f9ca494697dad0b4ab656547, State: Running, Role: LEADER
I20260927 05:07:14.267364 10277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc0c3f9ca494697dad0b4ab656547" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44967 } }
I20260927 05:07:14.268074 10272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:07:14.269908 10278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0abfc0c3f9ca494697dad0b4ab656547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc0c3f9ca494697dad0b4ab656547" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44967 } } }
I20260927 05:07:14.270215 10278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [sys.catalog]: This master's current role is: LEADER
I20260927 05:07:14.271227 10283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:07:14.271093 10279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0abfc0c3f9ca494697dad0b4ab656547. Latest consensus state: current_term: 1 leader_uuid: "0abfc0c3f9ca494697dad0b4ab656547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc0c3f9ca494697dad0b4ab656547" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44967 } } }
I20260927 05:07:14.271471 10279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547 [sys.catalog]: This master's current role is: LEADER
I20260927 05:07:14.277034 10283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:07:14.285530 10283 catalog_manager.cc:1347] Generated new cluster ID: 68a8ea5930d74620959dba4adc649b48
I20260927 05:07:14.285648 10283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:07:14.307096 10283 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:07:14.307528 10283 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:07:14.319988 10283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0abfc0c3f9ca494697dad0b4ab656547: Generated new TSK 0
I20260927 05:07:14.321080 10283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:07:14.332366 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44967
--builtin_ntp_servers=127.0.97.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 05:07:14.651557 10296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:14.651849 10296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:14.651988 10296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:14.659055 10296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:14.659147 10296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:14.659250 10296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:07:14.668419 10296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:14.669658 10296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:14.671373 10296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:14.685252 10301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:14.685278 10302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:14.685309 10304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:14.686259 10296 server_base.cc:1034] running on GCE node
I20260927 05:07:14.686765 10296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:14.687656 10296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:14.688850 10296 hybrid_clock.cc:648] HybridClock initialized: now 1790485634688775 us; error 65 us; skew 500 ppm
I20260927 05:07:14.689153 10296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:07:14.691303 10296 webserver.cc:460] Webserver started at http://127.0.97.1:32869/ using document root <none> and password file <none>
I20260927 05:07:14.692070 10296 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:14.692157 10296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:14.692462 10296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:14.695122 10296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "8055e500297d43f683dc4371cec0c03c"
format_stamp: "Formatted at 2026-09-27 05:07:14 on dist-test-slave-hgs0"
server_key: "bd403667c717ee0b13307a832630d316"
server_key_iv: "7199f46e7aa0096c72a52f61d4eb69de"
server_key_version: "encryptionkey@0"
I20260927 05:07:14.695873 10296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8055e500297d43f683dc4371cec0c03c"
format_stamp: "Formatted at 2026-09-27 05:07:14 on dist-test-slave-hgs0"
server_key: "bd403667c717ee0b13307a832630d316"
server_key_iv: "7199f46e7aa0096c72a52f61d4eb69de"
server_key_version: "encryptionkey@0"
I20260927 05:07:14.700783 10296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:07:14.704195 10310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:14.705493 10296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:07:14.705659 10296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "8055e500297d43f683dc4371cec0c03c"
format_stamp: "Formatted at 2026-09-27 05:07:14 on dist-test-slave-hgs0"
server_key: "bd403667c717ee0b13307a832630d316"
server_key_iv: "7199f46e7aa0096c72a52f61d4eb69de"
server_key_version: "encryptionkey@0"
I20260927 05:07:14.705919 10296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:14.711418 10296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:14.712350 10296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:14.712641 10296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:14.713572 10296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:07:14.714906 10296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:07:14.714993 10296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:14.715071 10296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:07:14.715124 10296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:14.746243 10296 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34781
I20260927 05:07:14.746287 10422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34781 every 8 connection(s)
I20260927 05:07:14.747579 10296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:07:14.753187 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 10296
I20260927 05:07:14.753397 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:07:14.753974 388 external_mini_cluster.cc:1428] Setting key 976a1c4ded3dc421391a50a90c1af93c
I20260927 05:07:14.755836 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:14.750301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46821 (local address 127.0.97.62:44967)
0927 05:07:14.750738 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:14.750742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:14.751837 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 05:07:14.753568 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:14.753574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:14.753649 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:14.753777 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:14.755094 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:14.755101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:14.755105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:14.755123 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:14.755170 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:14.755182 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:14.755311 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:14.755495 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 05:07:14.755557 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":235,"threads_started":1}
I20260927 05:07:14.756557 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44967
--builtin_ntp_servers=127.0.97.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 05:07:14.756842 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:14.750753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44967 (local address 127.0.97.1:46821)
0927 05:07:14.751689 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:14.751725 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:14.753290 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:14.754004 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:14.754035 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:14.754145 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:14.754818 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:07:14.754820 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:14.754855 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:07:14.754932 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:07:14.755276 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:07:14.755285 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:07:14.755293 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 05:07:14.755563 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 05:07:14.755792 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":263,"threads_started":1}
I20260927 05:07:14.758330 10423 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44967
I20260927 05:07:14.758626 10423 heartbeater.cc:464] Registering TS with master...
I20260927 05:07:14.759155 10423 heartbeater.cc:511] Master 127.0.97.62:44967 requested a full tablet report, sending...
I20260927 05:07:14.761220 10237 ts_manager.cc:194] Registered new tserver with Master: 8055e500297d43f683dc4371cec0c03c (127.0.97.1:34781)
W20260927 05:07:15.086294 10427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:15.086607 10427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:15.086769 10427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:15.094386 10427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:15.094555 10427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:15.094734 10427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:07:15.103910 10427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:15.105270 10427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:15.107070 10427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:15.120452 10433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:15.120435 10432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:15.120398 10435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:15.121387 10427 server_base.cc:1034] running on GCE node
I20260927 05:07:15.121960 10427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:15.122890 10427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:15.124038 10427 hybrid_clock.cc:648] HybridClock initialized: now 1790485635123984 us; error 33 us; skew 500 ppm
I20260927 05:07:15.124364 10427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:07:15.126629 10427 webserver.cc:460] Webserver started at http://127.0.97.2:46693/ using document root <none> and password file <none>
I20260927 05:07:15.127319 10427 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:15.127435 10427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:15.127756 10427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:15.130614 10427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "e401d91d0d8f435c80aadf7918aab3a1"
format_stamp: "Formatted at 2026-09-27 05:07:15 on dist-test-slave-hgs0"
server_key: "0c99617c5d21c87edcc1e66b26b32e07"
server_key_iv: "6a0192efb98229486981426fa0968986"
server_key_version: "encryptionkey@0"
I20260927 05:07:15.131410 10427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e401d91d0d8f435c80aadf7918aab3a1"
format_stamp: "Formatted at 2026-09-27 05:07:15 on dist-test-slave-hgs0"
server_key: "0c99617c5d21c87edcc1e66b26b32e07"
server_key_iv: "6a0192efb98229486981426fa0968986"
server_key_version: "encryptionkey@0"
I20260927 05:07:15.136598 10427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 05:07:15.139983 10441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:15.141366 10427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:07:15.141522 10427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "e401d91d0d8f435c80aadf7918aab3a1"
format_stamp: "Formatted at 2026-09-27 05:07:15 on dist-test-slave-hgs0"
server_key: "0c99617c5d21c87edcc1e66b26b32e07"
server_key_iv: "6a0192efb98229486981426fa0968986"
server_key_version: "encryptionkey@0"
I20260927 05:07:15.141729 10427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:15.147398 10427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:15.148228 10427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:15.148499 10427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:15.149430 10427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:07:15.150775 10427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:07:15.150889 10427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:15.151014 10427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:07:15.151109 10427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:15.186452 10427 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44997
I20260927 05:07:15.186507 10553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44997 every 8 connection(s)
I20260927 05:07:15.187773 10427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:07:15.189602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 10427
I20260927 05:07:15.189792 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:07:15.190323 388 external_mini_cluster.cc:1428] Setting key 26b34b56770be254f6ebcc410c99042d
I20260927 05:07:15.193373 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44967
--builtin_ntp_servers=127.0.97.20:33891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 05:07:15.196880 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:15.190846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54705 (local address 127.0.97.62:44967)
0927 05:07:15.191055 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:15.191061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:15.192775 (+ 1714us) server_negotiation.cc:407] Connection header received
0927 05:07:15.194556 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:15.194562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:15.194626 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:15.194755 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:15.196117 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:15.196122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:15.196127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:15.196145 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:15.196186 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:15.196201 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:15.196341 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:15.196521 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 05:07:15.196587 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:07:15.197774 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:15.191495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44967 (local address 127.0.97.2:54705)
0927 05:07:15.192598 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:15.192642 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:15.194298 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:15.194979 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:15.194987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:15.195079 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:15.195781 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:07:15.195783 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:15.195826 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:07:15.195908 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:07:15.196300 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:07:15.196310 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:07:15.196321 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 05:07:15.196605 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 05:07:15.196867 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":390,"threads_started":1}
I20260927 05:07:15.199318 10554 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44967
I20260927 05:07:15.199643 10554 heartbeater.cc:464] Registering TS with master...
I20260927 05:07:15.200418 10554 heartbeater.cc:511] Master 127.0.97.62:44967 requested a full tablet report, sending...
I20260927 05:07:15.202001 10237 ts_manager.cc:194] Registered new tserver with Master: e401d91d0d8f435c80aadf7918aab3a1 (127.0.97.2:44997)
W20260927 05:07:15.524547 10557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:15.524885 10557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:15.525058 10557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:15.532311 10557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:15.532464 10557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:15.532650 10557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:07:15.541661 10557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:15.542971 10557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:15.544751 10557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:15.557904 10562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:15.557992 10563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:15.558166 10565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:15.558826 10557 server_base.cc:1034] running on GCE node
I20260927 05:07:15.559423 10557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:15.560396 10557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:15.561583 10557 hybrid_clock.cc:648] HybridClock initialized: now 1790485635561524 us; error 57 us; skew 500 ppm
I20260927 05:07:15.561892 10557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:07:15.564090 10557 webserver.cc:460] Webserver started at http://127.0.97.3:35451/ using document root <none> and password file <none>
I20260927 05:07:15.564785 10557 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:15.564893 10557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:15.565219 10557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:15.568183 10557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "fdd25ff7077d484bbf85f7c139e5fe17"
format_stamp: "Formatted at 2026-09-27 05:07:15 on dist-test-slave-hgs0"
server_key: "900322b59b210b425aca79496fc33710"
server_key_iv: "476d22a7ed514250b406bf2940b86ba8"
server_key_version: "encryptionkey@0"
I20260927 05:07:15.568949 10557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fdd25ff7077d484bbf85f7c139e5fe17"
format_stamp: "Formatted at 2026-09-27 05:07:15 on dist-test-slave-hgs0"
server_key: "900322b59b210b425aca79496fc33710"
server_key_iv: "476d22a7ed514250b406bf2940b86ba8"
server_key_version: "encryptionkey@0"
I20260927 05:07:15.574033 10557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 05:07:15.577615 10571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:15.579028 10557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:07:15.579227 10557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "fdd25ff7077d484bbf85f7c139e5fe17"
format_stamp: "Formatted at 2026-09-27 05:07:15 on dist-test-slave-hgs0"
server_key: "900322b59b210b425aca79496fc33710"
server_key_iv: "476d22a7ed514250b406bf2940b86ba8"
server_key_version: "encryptionkey@0"
I20260927 05:07:15.579463 10557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:15.585404 10557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:15.586272 10557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:15.586557 10557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:15.587572 10557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:07:15.589054 10557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:07:15.589178 10557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:15.589318 10557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:07:15.589396 10557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:15.622807 10557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43555
I20260927 05:07:15.622849 10683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43555 every 8 connection(s)
I20260927 05:07:15.624327 10557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:07:15.627629 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 10557
I20260927 05:07:15.627835 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:07:15.628454 388 external_mini_cluster.cc:1428] Setting key ba29089fb10b216870e0536345e91d3a
I20260927 05:07:15.633319 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:15.626953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34163 (local address 127.0.97.62:44967)
0927 05:07:15.627159 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:15.627163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:15.628814 (+ 1651us) server_negotiation.cc:407] Connection header received
0927 05:07:15.631055 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:15.631060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:15.631150 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:15.631295 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:15.632789 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:15.632793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:15.632796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:15.632810 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:15.632839 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:15.632850 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:15.632936 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:15.633069 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 05:07:15.633113 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:07:15.634266 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:15.627444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44967 (local address 127.0.97.3:34163)
0927 05:07:15.628622 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:15.628680 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:15.630788 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:15.631579 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:15.631586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:15.631694 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:15.632522 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:07:15.632524 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:15.632561 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:07:15.632652 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:07:15.632913 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:07:15.632918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:07:15.632923 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 05:07:15.633118 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 05:07:15.633311 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":358,"threads_started":1}
I20260927 05:07:15.635465 10684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44967
I20260927 05:07:15.635762 10684 heartbeater.cc:464] Registering TS with master...
I20260927 05:07:15.636410 10684 heartbeater.cc:511] Master 127.0.97.62:44967 requested a full tablet report, sending...
I20260927 05:07:15.637863 10237 ts_manager.cc:194] Registered new tserver with Master: fdd25ff7077d484bbf85f7c139e5fe17 (127.0.97.3:43555)
I20260927 05:07:15.643532 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 05:07:15.650187 10425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:07:15.648486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.0.97.62:44967)
0927 05:07:15.648656 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:15.648660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:15.649167 (+ 507us) server_negotiation.cc:407] Connection header received
0927 05:07:15.649278 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:15.649283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:15.649348 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:15.649449 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:15.649868 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56208: BlockingRecv error: recv got EOF from 127.0.0.1:56208 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:07:15.650640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 10296
I20260927 05:07:15.654745 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:15.651903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.0.97.1:34781)
0927 05:07:15.652570 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:15.652586 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:15.652620 (+ 34us) server_negotiation.cc:407] Connection header received
0927 05:07:15.652704 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:15.652713 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:15.652971 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:15.653130 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:15.654074 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:15.654080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:15.654091 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:15.654102 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:15.654144 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:15.654156 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:15.654309 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:15.654404 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 05:07:15.654506 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":275,"threads_started":1}
I20260927 05:07:15.656044 10418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:15.737826 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 10296
I20260927 05:07:15.755252 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 10427
I20260927 05:07:15.758781 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:15.756238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37614 (local address 127.0.97.2:44997)
0927 05:07:15.756778 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:15.756785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:15.756815 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:07:15.756882 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:15.756889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:15.757135 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:15.757281 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:15.758120 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:15.758127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:15.758139 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:15.758159 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:15.758204 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:15.758220 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:15.758343 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:15.758414 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 05:07:15.758498 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":252,"threads_started":1}
I20260927 05:07:15.760195 10549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:15.840029 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 10427
I20260927 05:07:15.855804 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 10557
I20260927 05:07:15.859539 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:15.856917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.0.97.3:43555)
0927 05:07:15.857481 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:15.857487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:15.857520 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:07:15.857582 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:15.857588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:15.857825 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:15.857952 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:15.858925 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:15.858931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:15.858941 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:07:15.858952 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:15.858991 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:07:15.859008 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:15.859161 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:15.859211 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 05:07:15.859292 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":271,"threads_started":1}
I20260927 05:07:15.860760 10679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:15.940979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 10557
I20260927 05:07:15.956609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 10206
I20260927 05:07:15.957772 10267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:16.058095 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 10206
2026-09-27T05:07:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2297 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:07:16 dist-test-slave-hgs0 krb5kdc[10702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:07:16 dist-test-slave-hgs0 krb5kdc[10702](info): set up 2 sockets
Sep 27 05:07:16 dist-test-slave-hgs0 krb5kdc[10702](info): commencing operation
krb5kdc: starting...
W20260927 05:07:17.156724 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:07:17 dist-test-slave-hgs0 krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:07:17.392520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33557
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:07:17.718971 10715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:17.719283 10715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:17.719434 10715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:17.726680 10715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:07:17.726797 10715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:17.726893 10715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:17.726992 10715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:07:17.727063 10715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:07:17.736764 10715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:17.738135 10715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:17.739467 10715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:17.751284 10721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:17.751163 10723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:17.751529 10720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:17.752166 10715 server_base.cc:1034] running on GCE node
I20260927 05:07:17.752966 10715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:17.754565 10715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:17.755834 10715 hybrid_clock.cc:648] HybridClock initialized: now 1790485637755782 us; error 42 us; skew 500 ppm
I20260927 05:07:17.756287 10715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:07:17 dist-test-slave-hgs0 krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:07:17.763418 10715 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:07:17.765043 10715 webserver.cc:460] Webserver started at http://127.0.97.62:36407/ using document root <none> and password file <none>
I20260927 05:07:17.765722 10715 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:17.765823 10715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:17.766084 10715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:17.769119 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "db47ccb801f84bbcaff5c595cab8eede"
format_stamp: "Formatted at 2026-09-27 05:07:17 on dist-test-slave-hgs0"
server_key: "e1b8e8cc21fd877848fba046616d4e92"
server_key_iv: "cf0ea389b920f0da8e7bc2633eda25e1"
server_key_version: "encryptionkey@0"
I20260927 05:07:17.769833 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "db47ccb801f84bbcaff5c595cab8eede"
format_stamp: "Formatted at 2026-09-27 05:07:17 on dist-test-slave-hgs0"
server_key: "e1b8e8cc21fd877848fba046616d4e92"
server_key_iv: "cf0ea389b920f0da8e7bc2633eda25e1"
server_key_version: "encryptionkey@0"
I20260927 05:07:17.775445 10715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:07:17.779114 10730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:17.780644 10715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:07:17.780789 10715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "db47ccb801f84bbcaff5c595cab8eede"
format_stamp: "Formatted at 2026-09-27 05:07:17 on dist-test-slave-hgs0"
server_key: "e1b8e8cc21fd877848fba046616d4e92"
server_key_iv: "cf0ea389b920f0da8e7bc2633eda25e1"
server_key_version: "encryptionkey@0"
I20260927 05:07:17.780956 10715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:17.798849 10715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:17.803063 10715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:17.803303 10715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:17.821847 10715 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33557
I20260927 05:07:17.821883 10781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33557 every 8 connection(s)
I20260927 05:07:17.823176 10715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:07:17.826014 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 10715
I20260927 05:07:17.826256 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:07:17.827000 388 external_mini_cluster.cc:1428] Setting key cb92c2e60bd7ad5262d18a6c4b4764b8
I20260927 05:07:17.827582 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:07:17.841997 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede: Bootstrap starting.
Sep 27 05:07:17 dist-test-slave-hgs0 krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:07:17.845341 10782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede: Neither blocks nor log segments found. Creating new log.
I20260927 05:07:17.846485 10782 log.cc:824] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede: Log is configured to *not* fsync() on all Append() calls
I20260927 05:07:17.849290 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede: No bootstrap required, opened a new log
I20260927 05:07:17.850746 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:17.829098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.0.97.62:33557)
0927 05:07:17.829867 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:17.829879 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:17.829918 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:07:17.832134 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:17.832156 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:17.832558 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:17.832890 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:17.834762 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:17.836119 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:17.837827 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:17.838515 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:17.838649 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:17.843180 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:17.843189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:17.843207 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:17.843253 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:17.846356 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:17.847241 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:17.847257 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:17.847263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:17.847449 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:17.847874 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:17.847881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:17.847885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:17.848482 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:17.848801 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:17.849266 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 05:07:17.849704 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":369,"threads_started":1}
I20260927 05:07:17.857290 10782 raft_consensus.cc:348] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db47ccb801f84bbcaff5c595cab8eede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } }
I20260927 05:07:17.857681 10782 raft_consensus.cc:374] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:07:17.857793 10782 raft_consensus.cc:729] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db47ccb801f84bbcaff5c595cab8eede, State: Initialized, Role: FOLLOWER
I20260927 05:07:17.858335 10782 consensus_queue.cc:260] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db47ccb801f84bbcaff5c595cab8eede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } }
I20260927 05:07:17.858579 10782 raft_consensus.cc:388] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:07:17.858738 10782 raft_consensus.cc:482] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:07:17.858934 10782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:07:17.861114 10782 raft_consensus.cc:504] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db47ccb801f84bbcaff5c595cab8eede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } }
I20260927 05:07:17.861593 10782 leader_election.cc:302] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db47ccb801f84bbcaff5c595cab8eede; no voters:
I20260927 05:07:17.862170 10782 leader_election.cc:288] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:07:17.862295 10787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:07:17.863039 10787 raft_consensus.cc:686] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [term 1 LEADER]: Becoming Leader. State: Replica: db47ccb801f84bbcaff5c595cab8eede, State: Running, Role: LEADER
I20260927 05:07:17.863610 10787 consensus_queue.cc:237] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db47ccb801f84bbcaff5c595cab8eede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } }
I20260927 05:07:17.864480 10782 sys_catalog.cc:564] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:07:17.866570 10789 sys_catalog.cc:455] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [sys.catalog]: SysCatalogTable state changed. Reason: New leader db47ccb801f84bbcaff5c595cab8eede. Latest consensus state: current_term: 1 leader_uuid: "db47ccb801f84bbcaff5c595cab8eede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db47ccb801f84bbcaff5c595cab8eede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } } }
I20260927 05:07:17.866546 10788 sys_catalog.cc:455] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db47ccb801f84bbcaff5c595cab8eede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db47ccb801f84bbcaff5c595cab8eede" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33557 } } }
I20260927 05:07:17.866925 10789 sys_catalog.cc:458] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [sys.catalog]: This master's current role is: LEADER
I20260927 05:07:17.867005 10788 sys_catalog.cc:458] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede [sys.catalog]: This master's current role is: LEADER
I20260927 05:07:17.882584 10793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:07:17.889096 10793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:07:17.897675 10793 catalog_manager.cc:1347] Generated new cluster ID: fdf855bce4ef48388ed86677730180ca
I20260927 05:07:17.897862 10793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:07:17.906885 10793 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:07:17.908339 10793 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:07:17.930294 10793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede: Generated new TSK 0
I20260927 05:07:17.931205 10793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:07:18.001688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:07:18.349875 10808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:18.350226 10808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:18.350380 10808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:18.358140 10808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:18.358325 10808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:18.358495 10808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:07:18.369094 10808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:18.370486 10808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:18.371866 10808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:18.386650 10813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:18.386626 10814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:18.386763 10816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:18.387400 10808 server_base.cc:1034] running on GCE node
I20260927 05:07:18.388168 10808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:18.389228 10808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:18.390516 10808 hybrid_clock.cc:648] HybridClock initialized: now 1790485638390442 us; error 60 us; skew 500 ppm
I20260927 05:07:18.390905 10808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:07:18 dist-test-slave-hgs0 krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:07:18.394517 10808 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:07:18.396157 10808 webserver.cc:460] Webserver started at http://127.0.97.1:45577/ using document root <none> and password file <none>
I20260927 05:07:18.396925 10808 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:18.397073 10808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:18.397404 10808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:18.400449 10808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "9f90c90a5d3747e8922cf56758ce37b5"
format_stamp: "Formatted at 2026-09-27 05:07:18 on dist-test-slave-hgs0"
server_key: "ee4a8c1e2be299f4beaeedf1ae1fd4ac"
server_key_iv: "4c8e6413a28c5f3f30f0b159380254df"
server_key_version: "encryptionkey@0"
I20260927 05:07:18.401247 10808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9f90c90a5d3747e8922cf56758ce37b5"
format_stamp: "Formatted at 2026-09-27 05:07:18 on dist-test-slave-hgs0"
server_key: "ee4a8c1e2be299f4beaeedf1ae1fd4ac"
server_key_iv: "4c8e6413a28c5f3f30f0b159380254df"
server_key_version: "encryptionkey@0"
I20260927 05:07:18.406771 10808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:07:18.410568 10823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:18.412122 10808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:07:18.412334 10808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "9f90c90a5d3747e8922cf56758ce37b5"
format_stamp: "Formatted at 2026-09-27 05:07:18 on dist-test-slave-hgs0"
server_key: "ee4a8c1e2be299f4beaeedf1ae1fd4ac"
server_key_iv: "4c8e6413a28c5f3f30f0b159380254df"
server_key_version: "encryptionkey@0"
I20260927 05:07:18.412562 10808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:18.424109 10808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:18.428180 10808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:18.428507 10808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:18.429490 10808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:07:18.430871 10808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:07:18.430976 10808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:18.431135 10808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:07:18.431208 10808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:18.465080 10808 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40463
I20260927 05:07:18.465056 10935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40463 every 8 connection(s)
I20260927 05:07:18.466470 10808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:07:18.467684 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 10808
I20260927 05:07:18.467922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:07:18.468740 388 external_mini_cluster.cc:1428] Setting key c460a63401c8b3de9484c7db8435fe86
Sep 27 05:07:18 dist-test-slave-hgs0 krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:07:18.490780 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:18.469466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46559 (local address 127.0.97.62:33557)
0927 05:07:18.469944 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:18.469948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:18.471236 (+ 1288us) server_negotiation.cc:407] Connection header received
0927 05:07:18.473495 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:18.473502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:18.473594 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:18.473715 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:18.475289 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:18.476502 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:18.478179 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:18.478925 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:18.479075 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:18.482913 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:18.482923 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:18.482931 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:18.482979 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:18.486825 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:18.487694 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:18.487703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:18.487709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:18.487869 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:18.488458 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:18.488466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:18.488472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:18.488894 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:18.489059 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:18.489971 (+ 912us) server_negotiation.cc:299] Negotiation successful
0927 05:07:18.490259 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
I20260927 05:07:18.491451 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:18.470097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33557 (local address 127.0.97.1:46559)
0927 05:07:18.471057 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:18.471109 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:18.473185 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:18.473985 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:18.473994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:18.474518 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:18.475106 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:18.475124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:18.476674 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:18.476678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:18.478014 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:18.478026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:18.479144 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:18.479149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:18.479366 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:18.480108 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:18.480130 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:18.482645 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:07:18.487013 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:07:18.487022 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:07:18.487038 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:07:18.487551 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:07:18.488100 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:07:18.488104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:07:18.488106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:07:18.488325 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:07:18.489112 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:07:18.489120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:07:18.489537 (+ 417us) client_negotiation.cc:769] Sending connection context
0927 05:07:18.489806 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 05:07:18.490257 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":394,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:07:18.493268 10936 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33557
I20260927 05:07:18.493661 10936 heartbeater.cc:464] Registering TS with master...
I20260927 05:07:18.494374 10936 heartbeater.cc:511] Master 127.0.97.62:33557 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:07:18.496994 10747 ts_manager.cc:194] Registered new tserver with Master: 9f90c90a5d3747e8922cf56758ce37b5 (127.0.97.1:40463)
I20260927 05:07:18.498622 10747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:07:18.528502 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:07:18.885290 10942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:18.885649 10942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:18.885814 10942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:18.898137 10942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:18.898324 10942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:18.898715 10942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:07:18.916404 10942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:18.918201 10942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:18.920125 10942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:18.938861 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:18.938861 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:18.939815 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:18.940480 10942 server_base.cc:1034] running on GCE node
I20260927 05:07:18.941437 10942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:18.942713 10942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:18.944047 10942 hybrid_clock.cc:648] HybridClock initialized: now 1790485638943956 us; error 78 us; skew 500 ppm
I20260927 05:07:18.944458 10942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:07:18 dist-test-slave-hgs0 krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:07:18.948331 10942 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:07:18.950037 10942 webserver.cc:460] Webserver started at http://127.0.97.2:36901/ using document root <none> and password file <none>
I20260927 05:07:18.950781 10942 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:18.950886 10942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:18.951159 10942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:18.954348 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "fd6b75c7156d47e2a50d324c6665af37"
format_stamp: "Formatted at 2026-09-27 05:07:18 on dist-test-slave-hgs0"
server_key: "3038b094e5772a51c920af4827df712a"
server_key_iv: "398e48fdd7a3f73c1cda5fb22399e2aa"
server_key_version: "encryptionkey@0"
I20260927 05:07:18.955145 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fd6b75c7156d47e2a50d324c6665af37"
format_stamp: "Formatted at 2026-09-27 05:07:18 on dist-test-slave-hgs0"
server_key: "3038b094e5772a51c920af4827df712a"
server_key_iv: "398e48fdd7a3f73c1cda5fb22399e2aa"
server_key_version: "encryptionkey@0"
I20260927 05:07:18.961098 10942 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 05:07:18.965273 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:18.967013 10942 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:07:18.967216 10942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "fd6b75c7156d47e2a50d324c6665af37"
format_stamp: "Formatted at 2026-09-27 05:07:18 on dist-test-slave-hgs0"
server_key: "3038b094e5772a51c920af4827df712a"
server_key_iv: "398e48fdd7a3f73c1cda5fb22399e2aa"
server_key_version: "encryptionkey@0"
I20260927 05:07:18.967480 10942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:18.998262 10942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:19.003355 10942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:19.003713 10942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:19.004972 10942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:07:19.006731 10942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:07:19.006847 10942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:19.006995 10942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:07:19.007078 10942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:19.042846 10942 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43845
I20260927 05:07:19.042951 11069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43845 every 8 connection(s)
I20260927 05:07:19.044373 10942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:07:19.054342 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 10942
I20260927 05:07:19.054560 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:07:19.054982 388 external_mini_cluster.cc:1428] Setting key 1a129abecf5d007be30a85620df55b00
Sep 27 05:07:19 dist-test-slave-hgs0 krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:07:19.066211 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.047225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43199 (local address 127.0.97.62:33557)
0927 05:07:19.047686 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:19.047692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:19.048929 (+ 1237us) server_negotiation.cc:407] Connection header received
0927 05:07:19.050769 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:19.050773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:19.050853 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:19.050960 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:19.052658 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.053369 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.055108 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.055706 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.055814 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.059779 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:19.059784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:19.059788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:19.059824 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:19.062740 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:19.063625 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:19.063629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:19.063632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:19.063763 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:19.064308 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:19.064312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:19.064314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:19.064614 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:19.064805 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:19.065577 (+ 772us) server_negotiation.cc:299] Negotiation successful
0927 05:07:19.065811 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260927 05:07:19.067329 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.048005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33557 (local address 127.0.97.2:43199)
0927 05:07:19.048739 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:19.048785 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:19.050441 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:19.051195 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:19.051204 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:19.051741 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:19.052467 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.052486 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.053515 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.053518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.054940 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.054953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.056060 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.056068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.056331 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.057066 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:19.057090 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:19.059504 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:07:19.062943 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:07:19.062954 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:07:19.062977 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:07:19.063466 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:07:19.063886 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:07:19.063892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:07:19.063896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:07:19.064161 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:07:19.064800 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:07:19.064810 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:07:19.065197 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 05:07:19.065477 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 05:07:19.066117 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":302,"threads_started":1}
I20260927 05:07:19.068958 11070 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33557
I20260927 05:07:19.069326 11070 heartbeater.cc:464] Registering TS with master...
I20260927 05:07:19.070261 11070 heartbeater.cc:511] Master 127.0.97.62:33557 requested a full tablet report, sending...
I20260927 05:07:19.072198 10747 ts_manager.cc:194] Registered new tserver with Master: fd6b75c7156d47e2a50d324c6665af37 (127.0.97.2:43845)
I20260927 05:07:19.073604 10747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43199
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:07:19.112704 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:07:19.483595 11076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:07:19.483901 11076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:07:19.484097 11076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:07:19.491677 11076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:07:19.491815 11076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:07:19.492057 11076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:07:19.501941 10936 heartbeater.cc:503] Master 127.0.97.62:33557 was elected leader, sending a full tablet report...
I20260927 05:07:19.501935 11076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:07:19.503607 11076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:07:19.505076 11076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:07:19.519374 11084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:19.519325 11081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:07:19.519420 11082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:07:19.520517 11076 server_base.cc:1034] running on GCE node
I20260927 05:07:19.521337 11076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:07:19.522550 11076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:07:19.523901 11076 hybrid_clock.cc:648] HybridClock initialized: now 1790485639523833 us; error 58 us; skew 500 ppm
I20260927 05:07:19.524448 11076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:07:19 dist-test-slave-hgs0 krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:07:19.528581 11076 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:07:19.530264 11076 webserver.cc:460] Webserver started at http://127.0.97.3:46227/ using document root <none> and password file <none>
I20260927 05:07:19.531028 11076 fs_manager.cc:359] Metadata directory not provided
I20260927 05:07:19.531136 11076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:07:19.531432 11076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:07:19.534617 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "676265283b704837b96c1f19929abd39"
format_stamp: "Formatted at 2026-09-27 05:07:19 on dist-test-slave-hgs0"
server_key: "b1c1f53fab5b316c79c1e45a5244d909"
server_key_iv: "84bdc4566b572819169eafc5840df559"
server_key_version: "encryptionkey@0"
I20260927 05:07:19.535451 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "676265283b704837b96c1f19929abd39"
format_stamp: "Formatted at 2026-09-27 05:07:19 on dist-test-slave-hgs0"
server_key: "b1c1f53fab5b316c79c1e45a5244d909"
server_key_iv: "84bdc4566b572819169eafc5840df559"
server_key_version: "encryptionkey@0"
I20260927 05:07:19.541121 11076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 05:07:19.544947 11091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:19.546555 11076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:07:19.546733 11076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "676265283b704837b96c1f19929abd39"
format_stamp: "Formatted at 2026-09-27 05:07:19 on dist-test-slave-hgs0"
server_key: "b1c1f53fab5b316c79c1e45a5244d909"
server_key_iv: "84bdc4566b572819169eafc5840df559"
server_key_version: "encryptionkey@0"
I20260927 05:07:19.546936 11076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:07:19.572592 11076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:07:19.577497 11076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:07:19.577759 11076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:07:19.578747 11076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:07:19.580207 11076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:07:19.580286 11076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:19.580375 11076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:07:19.580421 11076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:07:19.619200 11076 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38411
I20260927 05:07:19.619306 11203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38411 every 8 connection(s)
I20260927 05:07:19.620735 11076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:07:19.626232 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 11076
I20260927 05:07:19.626484 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:07:19.626915 388 external_mini_cluster.cc:1428] Setting key 9bebdf1581711b4653ebce70786ef323
Sep 27 05:07:19 dist-test-slave-hgs0 krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:07:19.643304 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.623841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36425 (local address 127.0.97.62:33557)
0927 05:07:19.624431 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:19.624436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:19.625538 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 05:07:19.627706 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:19.627714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:19.627821 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:19.627993 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:19.629956 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.630810 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.632569 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.632989 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.633097 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.636942 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:19.636947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:19.636951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:19.636990 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:19.639531 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:19.640485 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:19.640491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:19.640502 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:19.640639 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:19.641176 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:19.641181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:19.641185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:19.641551 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:19.641727 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:19.642650 (+ 923us) server_negotiation.cc:299] Negotiation successful
0927 05:07:19.642908 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":323,"threads_started":1}
I20260927 05:07:19.644524 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.624550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33557 (local address 127.0.97.3:36425)
0927 05:07:19.625346 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:19.625390 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:19.627379 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:19.628360 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:19.628371 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:19.629029 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:07:19.629754 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.629783 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.630962 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.630967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.632413 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.632429 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.633200 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.633205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.633437 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.634227 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:07:19.634253 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:07:19.636648 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:07:19.639700 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:07:19.639709 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:07:19.639725 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:07:19.640318 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:07:19.640772 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:07:19.640777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:07:19.640782 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:07:19.641057 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:07:19.641704 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:07:19.641713 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:07:19.642208 (+ 495us) client_negotiation.cc:769] Sending connection context
0927 05:07:19.642527 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 05:07:19.643133 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":307,"threads_started":1}
I20260927 05:07:19.646128 11204 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33557
I20260927 05:07:19.646466 11204 heartbeater.cc:464] Registering TS with master...
I20260927 05:07:19.647246 11204 heartbeater.cc:511] Master 127.0.97.62:33557 requested a full tablet report, sending...
I20260927 05:07:19.648913 10747 ts_manager.cc:194] Registered new tserver with Master: 676265283b704837b96c1f19929abd39 (127.0.97.3:38411)
I20260927 05:07:19.649758 10747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36425
I20260927 05:07:19.655205 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:07:19.674402 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.660049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.0.97.62:33557)
0927 05:07:19.660288 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:19.660293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:19.660831 (+ 538us) server_negotiation.cc:407] Connection header received
0927 05:07:19.661002 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:19.661006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:19.661074 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:19.661174 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:19.662693 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.663542 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.665138 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.665725 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.665846 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.668463 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:19.668468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:19.668472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:19.668510 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:19.671323 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:19.672239 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:19.672244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:19.672246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:19.672377 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:19.672794 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:19.672797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:19.672800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:19.673189 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:19.673492 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:19.673832 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 05:07:19.674067 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 05:07:19.679184 10747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:07:19.687084 10747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:07:19.715576 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.700928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43845 (local address 127.0.0.1:51978)
0927 05:07:19.702590 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:19.702649 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:19.702916 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:19.704273 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:19.704283 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:19.704316 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:07:19.704801 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.704817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.707671 (+ 2854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.707677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.711152 (+ 3475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.711184 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.714252 (+ 3068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.714265 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.714543 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.714610 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 05:07:19.714798 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 05:07:19.715094 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"thread_start_us":1206,"threads_started":1}
I20260927 05:07:19.716424 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.701147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51978 (local address 127.0.97.2:43845)
0927 05:07:19.702114 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:19.702127 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:19.703187 (+ 1060us) server_negotiation.cc:407] Connection header received
0927 05:07:19.703339 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:19.703352 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:19.703695 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:19.703929 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:07:19.705042 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.707410 (+ 2368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.711493 (+ 4083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.713539 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.713699 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.713982 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:19.715788 (+ 1806us) server_negotiation.cc:299] Negotiation successful
0927 05:07:19.716038 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":622,"threads_started":1}
I20260927 05:07:19.721125 11005 tablet_service.cc:1467] Processing CreateTablet for tablet 112312b0e0d94b5e8ce44e975339c0df (DEFAULT_TABLE table=test-table [id=17f1b12b159244d99dba86c3404964d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:07:19.722733 11005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112312b0e0d94b5e8ce44e975339c0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:07:19.724470 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.702476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.0.97.1:40463)
0927 05:07:19.712494 (+ 10018us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:19.712507 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:19.715532 (+ 3025us) server_negotiation.cc:407] Connection header received
0927 05:07:19.715668 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:19.715680 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:19.716203 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:19.716455 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:07:19.717341 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.719236 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.721568 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.723105 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.723281 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.723550 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:19.723788 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 05:07:19.724039 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9760,"thread_start_us":11024,"threads_started":1}
I20260927 05:07:19.725652 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.701532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40463 (local address 127.0.0.1:33120)
0927 05:07:19.715206 (+ 13674us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:19.715247 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:19.715386 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:19.716434 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:19.716439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:19.716468 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:07:19.717082 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.717095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.719421 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.719427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.721366 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.721382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.723367 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.723377 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.723561 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.723598 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:07:19.725023 (+ 1425us) client_negotiation.cc:240] Negotiation successful
0927 05:07:19.725199 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13524,"mutex_wait_us":1,"thread_start_us":12429,"threads_started":1}
I20260927 05:07:19.725709 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.702131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.0.97.3:38411)
0927 05:07:19.716168 (+ 14037us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:19.716181 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:19.716220 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:07:19.716342 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:19.716351 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:19.716642 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:19.716942 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:07:19.717781 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.719506 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.722320 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.724033 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.724185 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.724466 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:19.725137 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 05:07:19.725326 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13481,"thread_start_us":13265,"threads_started":1}
I20260927 05:07:19.726414 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.701363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38411 (local address 127.0.0.1:48328)
0927 05:07:19.714427 (+ 13064us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:19.714459 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:19.714598 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:19.717041 (+ 2443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:19.717046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:19.717080 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:07:19.717547 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.717560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.719691 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.719697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.722125 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.722140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.724640 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.724648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.724887 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.724923 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:07:19.725020 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 05:07:19.726093 (+ 1073us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12902,"mutex_wait_us":878,"thread_start_us":13532,"threads_started":1}
I20260927 05:07:19.729844 10871 tablet_service.cc:1467] Processing CreateTablet for tablet 112312b0e0d94b5e8ce44e975339c0df (DEFAULT_TABLE table=test-table [id=17f1b12b159244d99dba86c3404964d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:07:19.730429 11139 tablet_service.cc:1467] Processing CreateTablet for tablet 112312b0e0d94b5e8ce44e975339c0df (DEFAULT_TABLE table=test-table [id=17f1b12b159244d99dba86c3404964d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:07:19.731531 10871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112312b0e0d94b5e8ce44e975339c0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:07:19.732676 11139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112312b0e0d94b5e8ce44e975339c0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:07:19.738636 11223 tablet_bootstrap.cc:492] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: Bootstrap starting.
I20260927 05:07:19.743124 11223 tablet_bootstrap.cc:654] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: Neither blocks nor log segments found. Creating new log.
I20260927 05:07:19.744702 11223 log.cc:824] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: Log is configured to *not* fsync() on all Append() calls
I20260927 05:07:19.746348 11225 tablet_bootstrap.cc:492] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: Bootstrap starting.
I20260927 05:07:19.748343 11226 tablet_bootstrap.cc:492] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: Bootstrap starting.
I20260927 05:07:19.748728 11223 tablet_bootstrap.cc:492] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: No bootstrap required, opened a new log
I20260927 05:07:19.749037 11223 ts_tablet_manager.cc:1397] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 05:07:19.750587 11225 tablet_bootstrap.cc:654] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: Neither blocks nor log segments found. Creating new log.
I20260927 05:07:19.752162 11225 log.cc:824] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: Log is configured to *not* fsync() on all Append() calls
I20260927 05:07:19.752360 11226 tablet_bootstrap.cc:654] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: Neither blocks nor log segments found. Creating new log.
I20260927 05:07:19.754060 11226 log.cc:824] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: Log is configured to *not* fsync() on all Append() calls
I20260927 05:07:19.757486 11225 tablet_bootstrap.cc:492] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: No bootstrap required, opened a new log
I20260927 05:07:19.757802 11226 tablet_bootstrap.cc:492] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: No bootstrap required, opened a new log
I20260927 05:07:19.757855 11225 ts_tablet_manager.cc:1397] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260927 05:07:19.758234 11226 ts_tablet_manager.cc:1397] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260927 05:07:19.759231 11223 raft_consensus.cc:348] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6b75c7156d47e2a50d324c6665af37" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43845 } } peers { permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 } } peers { permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 } }
I20260927 05:07:19.759749 11223 raft_consensus.cc:374] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:07:19.759873 11223 raft_consensus.cc:729] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd6b75c7156d47e2a50d324c6665af37, State: Initialized, Role: FOLLOWER
I20260927 05:07:19.760589 11223 consensus_queue.cc:260] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6b75c7156d47e2a50d324c6665af37" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43845 } } peers { permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 } } peers { permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 } }
I20260927 05:07:19.761942 11070 heartbeater.cc:503] Master 127.0.97.62:33557 was elected leader, sending a full tablet report...
I20260927 05:07:19.762650 11223 ts_tablet_manager.cc:1428] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260927 05:07:19.764111 11229 raft_consensus.cc:482] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:07:19.764446 11229 raft_consensus.cc:504] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6b75c7156d47e2a50d324c6665af37" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43845 } } peers { permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 } } peers { permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 } }
I20260927 05:07:19.769142 11225 raft_consensus.cc:348] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6b75c7156d47e2a50d324c6665af37" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43845 } } peers { permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 } } peers { permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 } }
I20260927 05:07:19.769706 11225 raft_consensus.cc:374] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:07:19.769846 11225 raft_consensus.cc:729] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f90c90a5d3747e8922cf56758ce37b5, State: Initialized, Role: FOLLOWER
I20260927 05:07:19.770603 11225 consensus_queue.cc:260] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6b75c7156d47e2a50d324c6665af37" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43845 } } peers { permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 } } peers { permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 } }
I20260927 05:07:19.771353 11226 raft_consensus.cc:348] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6b75c7156d47e2a50d324c6665af37" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43845 } } peers { permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 } } peers { permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 } }
I20260927 05:07:19.772010 11226 raft_consensus.cc:374] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:07:19.772142 11226 raft_consensus.cc:729] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676265283b704837b96c1f19929abd39, State: Initialized, Role: FOLLOWER
I20260927 05:07:19.773020 11229 leader_election.cc:288] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 676265283b704837b96c1f19929abd39 (127.0.97.3:38411), 9f90c90a5d3747e8922cf56758ce37b5 (127.0.97.1:40463)
I20260927 05:07:19.773056 11226 consensus_queue.cc:260] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6b75c7156d47e2a50d324c6665af37" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43845 } } peers { permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 } } peers { permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 } }
I20260927 05:07:19.774499 11204 heartbeater.cc:503] Master 127.0.97.62:33557 was elected leader, sending a full tablet report...
I20260927 05:07:19.775337 11226 ts_tablet_manager.cc:1428] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: Time spent starting tablet: real 0.017s user 0.007s sys 0.008s
I20260927 05:07:19.778538 11225 ts_tablet_manager.cc:1428] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: Time spent starting tablet: real 0.021s user 0.007s sys 0.010s
I20260927 05:07:19.783200 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.772446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46529 (local address 127.0.97.1:40463)
0927 05:07:19.772651 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:19.772659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:19.773935 (+ 1276us) server_negotiation.cc:407] Connection header received
0927 05:07:19.774263 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:19.774269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:19.774359 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:19.774425 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:07:19.775648 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.777072 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.780990 (+ 3918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.782187 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.782355 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.782473 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:19.782808 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 05:07:19.782935 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:07:19.783592 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.771760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40463 (local address 127.0.97.2:46529)
0927 05:07:19.773834 (+ 2074us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:19.773861 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:19.774081 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:19.774635 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:19.774646 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:19.774669 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:07:19.775349 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.775370 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.777715 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.777720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.780810 (+ 3090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.780828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.782431 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.782437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.782633 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.782667 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:07:19.782770 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 05:07:19.782922 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1881,"mutex_wait_us":70,"thread_start_us":892,"threads_started":1}
I20260927 05:07:19.784202 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.766958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38411 (local address 127.0.97.2:57937)
0927 05:07:19.772680 (+ 5722us) negotiation.cc:107] Waiting for socket to connect
0927 05:07:19.772703 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:07:19.773540 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:07:19.774428 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:07:19.774432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:07:19.774458 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:07:19.774926 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.774940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.778956 (+ 4016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.778963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.781441 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:07:19.781455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.783243 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.783251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:07:19.783482 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.783521 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 05:07:19.783643 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 05:07:19.783796 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5558,"thread_start_us":6543,"threads_started":1}
I20260927 05:07:19.784744 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112312b0e0d94b5e8ce44e975339c0df" candidate_uuid: "fd6b75c7156d47e2a50d324c6665af37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f90c90a5d3747e8922cf56758ce37b5" is_pre_election: true
I20260927 05:07:19.785305 10891 raft_consensus.cc:2454] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd6b75c7156d47e2a50d324c6665af37 in term 0.
I20260927 05:07:19.785893 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.767443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57937 (local address 127.0.97.3:38411)
0927 05:07:19.767657 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:19.767665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:19.772820 (+ 5155us) server_negotiation.cc:407] Connection header received
0927 05:07:19.773981 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:19.773988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:19.774089 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:19.774167 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:07:19.775975 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.777158 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.781642 (+ 4484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.782987 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.783160 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.783275 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:19.783697 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 05:07:19.783855 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:07:19.786128 10960 leader_election.cc:302] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f90c90a5d3747e8922cf56758ce37b5, fd6b75c7156d47e2a50d324c6665af37; no voters:
I20260927 05:07:19.786639 11229 raft_consensus.cc:2781] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:07:19.786835 11229 raft_consensus.cc:482] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:07:19.786972 11229 raft_consensus.cc:3037] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:07:19.787103 11159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112312b0e0d94b5e8ce44e975339c0df" candidate_uuid: "fd6b75c7156d47e2a50d324c6665af37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "676265283b704837b96c1f19929abd39" is_pre_election: true
I20260927 05:07:19.787627 11159 raft_consensus.cc:2454] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd6b75c7156d47e2a50d324c6665af37 in term 0.
I20260927 05:07:19.789775 11229 raft_consensus.cc:504] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6b75c7156d47e2a50d324c6665af37" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43845 } } peers { permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 } } peers { permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 } }
I20260927 05:07:19.790500 11229 leader_election.cc:288] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [CANDIDATE]: Term 1 election: Requested vote from peers 676265283b704837b96c1f19929abd39 (127.0.97.3:38411), 9f90c90a5d3747e8922cf56758ce37b5 (127.0.97.1:40463)
I20260927 05:07:19.791117 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112312b0e0d94b5e8ce44e975339c0df" candidate_uuid: "fd6b75c7156d47e2a50d324c6665af37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f90c90a5d3747e8922cf56758ce37b5"
I20260927 05:07:19.791199 11159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112312b0e0d94b5e8ce44e975339c0df" candidate_uuid: "fd6b75c7156d47e2a50d324c6665af37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "676265283b704837b96c1f19929abd39"
I20260927 05:07:19.791391 10891 raft_consensus.cc:3037] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:07:19.791419 11159 raft_consensus.cc:3037] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:07:19.795298 11159 raft_consensus.cc:2454] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd6b75c7156d47e2a50d324c6665af37 in term 1.
I20260927 05:07:19.795298 10891 raft_consensus.cc:2454] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd6b75c7156d47e2a50d324c6665af37 in term 1.
I20260927 05:07:19.795926 10960 leader_election.cc:302] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f90c90a5d3747e8922cf56758ce37b5, fd6b75c7156d47e2a50d324c6665af37; no voters:
I20260927 05:07:19.796469 11229 raft_consensus.cc:2781] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 1 FOLLOWER]: Leader election won for term 1
W20260927 05:07:19.796489 11071 tablet.cc:2369] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:07:19.797065 11229 raft_consensus.cc:686] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 1 LEADER]: Becoming Leader. State: Replica: fd6b75c7156d47e2a50d324c6665af37, State: Running, Role: LEADER
I20260927 05:07:19.797701 11229 consensus_queue.cc:237] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6b75c7156d47e2a50d324c6665af37" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43845 } } peers { permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 } } peers { permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 } }
I20260927 05:07:19.803135 10746 catalog_manager.cc:5495] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd6b75c7156d47e2a50d324c6665af37 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fd6b75c7156d47e2a50d324c6665af37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd6b75c7156d47e2a50d324c6665af37" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 } health_report { overall_health: UNKNOWN } } }
I20260927 05:07:19.841037 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.835065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.0.97.2:43845)
0927 05:07:19.835279 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:19.835286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:19.835308 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:07:19.835506 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:19.835512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:19.835616 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:19.835777 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:07:19.836199 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.837347 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.838746 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.839159 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.839259 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.839767 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:07:19.840420 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:07:19.840546 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:19.840637 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 05:07:19.840718 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:07:19.849905 10891 raft_consensus.cc:1261] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5 [term 1 FOLLOWER]: Refusing update from remote peer fd6b75c7156d47e2a50d324c6665af37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:07:19.849905 11159 raft_consensus.cc:1261] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39 [term 1 FOLLOWER]: Refusing update from remote peer fd6b75c7156d47e2a50d324c6665af37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:07:19.850922 11229 consensus_queue.cc:1035] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f90c90a5d3747e8922cf56758ce37b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:07:19.851241 11234 consensus_queue.cc:1035] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "676265283b704837b96c1f19929abd39" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:07:19.864058 11237 mvcc.cc:205] Tried to move back new op lower bound from 7333829180814274560 to 7333829180616122368. Current Snapshot: MvccSnapshot[applied={T|T < 7333829180814274560}]
I20260927 05:07:19.880666 10746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:07:19.880983 10746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:07:19.884919 10746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db47ccb801f84bbcaff5c595cab8eede: Sending DeleteTablet for 3 replicas of tablet 112312b0e0d94b5e8ce44e975339c0df
I20260927 05:07:19.886351 11139 tablet_service.cc:1514] Processing DeleteTablet for tablet 112312b0e0d94b5e8ce44e975339c0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:07:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48328
I20260927 05:07:19.886622 11005 tablet_service.cc:1514] Processing DeleteTablet for tablet 112312b0e0d94b5e8ce44e975339c0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:07:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51978
I20260927 05:07:19.887302 10871 tablet_service.cc:1514] Processing DeleteTablet for tablet 112312b0e0d94b5e8ce44e975339c0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:07:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33120
I20260927 05:07:19.888254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 10808
I20260927 05:07:19.900633 11249 tablet_replica.cc:330] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: stopping tablet replica
I20260927 05:07:19.900940 11247 tablet_replica.cc:330] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: stopping tablet replica
I20260927 05:07:19.901175 11249 raft_consensus.cc:2229] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:07:19.901458 11247 raft_consensus.cc:2229] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:07:19.901759 11249 raft_consensus.cc:2258] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:07:19.902009 11247 raft_consensus.cc:2258] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:07:19.904650 11247 ts_tablet_manager.cc:1905] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:07:19.906265 11249 ts_tablet_manager.cc:1905] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:07:19.911865 11248 tablet_replica.cc:330] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: stopping tablet replica
I20260927 05:07:19.912953 11247 ts_tablet_manager.cc:1918] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:07:19.913158 11247 log.cc:1196] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/112312b0e0d94b5e8ce44e975339c0df
I20260927 05:07:19.913028 11248 raft_consensus.cc:2229] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:07:19.913815 11247 ts_tablet_manager.cc:1939] T 112312b0e0d94b5e8ce44e975339c0df P 676265283b704837b96c1f19929abd39: Deleting consensus metadata
I20260927 05:07:19.914222 11248 raft_consensus.cc:2258] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:07:19.916296 10731 catalog_manager.cc:4841] TS 676265283b704837b96c1f19929abd39 (127.0.97.3:38411): tablet 112312b0e0d94b5e8ce44e975339c0df (table test-table [id=17f1b12b159244d99dba86c3404964d5]) successfully deleted
Sep 27 05:07:19 dist-test-slave-hgs0 krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:07:19.918515 11248 ts_tablet_manager.cc:1905] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:07:19.918920 11249 ts_tablet_manager.cc:1918] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:07:19.919097 11249 log.cc:1196] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/112312b0e0d94b5e8ce44e975339c0df
I20260927 05:07:19.919683 11249 ts_tablet_manager.cc:1939] T 112312b0e0d94b5e8ce44e975339c0df P 9f90c90a5d3747e8922cf56758ce37b5: Deleting consensus metadata
I20260927 05:07:19.922101 10733 catalog_manager.cc:4841] TS 9f90c90a5d3747e8922cf56758ce37b5 (127.0.97.1:40463): tablet 112312b0e0d94b5e8ce44e975339c0df (table test-table [id=17f1b12b159244d99dba86c3404964d5]) successfully deleted
I20260927 05:07:19.926456 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:19.890273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.0.97.1:40463)
0927 05:07:19.890532 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:19.890543 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:19.895518 (+ 4975us) server_negotiation.cc:407] Connection header received
0927 05:07:19.896830 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:19.896841 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:19.897006 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:19.897111 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:19.903998 (+ 6887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.905280 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.910638 (+ 5358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:19.911623 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:19.911787 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:19.919669 (+ 7882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:19.919684 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:19.919702 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:19.919759 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:19.923368 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:19.924294 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:19.924311 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:19.924315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:19.924450 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:19.924856 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:19.924861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:19.924871 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:19.925215 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:19.925532 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:19.925832 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 05:07:19.926081 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:07:19.927155 10931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:19.927172 11248 ts_tablet_manager.cc:1918] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:07:19.927388 11248 log.cc:1196] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/112312b0e0d94b5e8ce44e975339c0df
I20260927 05:07:19.928105 11248 ts_tablet_manager.cc:1939] T 112312b0e0d94b5e8ce44e975339c0df P fd6b75c7156d47e2a50d324c6665af37: Deleting consensus metadata
I20260927 05:07:19.930266 10734 catalog_manager.cc:4841] TS fd6b75c7156d47e2a50d324c6665af37 (127.0.97.2:43845): tablet 112312b0e0d94b5e8ce44e975339c0df (table test-table [id=17f1b12b159244d99dba86c3404964d5]) successfully deleted
I20260927 05:07:20.080130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 10808
I20260927 05:07:20.099879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 10942
Sep 27 05:07:20 dist-test-slave-hgs0 krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:07:20.119541 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:20.100832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.0.97.2:43845)
0927 05:07:20.101076 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:20.101083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:20.101105 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:07:20.101328 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:20.101334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:20.101422 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:20.101547 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:20.102922 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:20.104243 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:20.105627 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:20.106443 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:20.106587 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:20.110914 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:20.110936 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:20.110959 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:20.111022 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:20.115585 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:20.116638 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:20.116651 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:20.116656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:20.116886 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:20.117434 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:20.117442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:20.117446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:20.118010 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:20.118409 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:20.118769 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 05:07:20.119057 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 05:07:20.120690 11065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:20.245443 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 10942
I20260927 05:07:20.267479 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 11076
Sep 27 05:07:20 dist-test-slave-hgs0 krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:07:20.284276 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:07:20.268456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.0.97.3:38411)
0927 05:07:20.268645 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 05:07:20.268651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:07:20.268672 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:07:20.268788 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:07:20.268793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:07:20.268859 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:07:20.268946 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:07:20.270300 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:20.271083 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:20.272701 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:07:20.273143 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:07:20.273273 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:07:20.278144 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:07:20.278163 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:07:20.278186 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:07:20.278232 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:07:20.281217 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:20.281965 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:20.281975 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:20.281977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:20.282147 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:07:20.282548 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:07:20.282553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:07:20.282555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:07:20.283022 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:07:20.283309 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 05:07:20.283567 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 05:07:20.283773 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 05:07:20.285081 11199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:20.379644 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 11076
I20260927 05:07:20.396996 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 10715
I20260927 05:07:20.398233 10777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:07:20.509459 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 10715
2026-09-27T05:07:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4458 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:07:20 dist-test-slave-hgs0 krb5kdc[11257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:07:20 dist-test-slave-hgs0 krb5kdc[11257](info): set up 2 sockets
Sep 27 05:07:20 dist-test-slave-hgs0 krb5kdc[11257](info): commencing operation
krb5kdc: starting...
W20260927 05:07:21.660869 388 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:07:21 dist-test-slave-hgs0 krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:07:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:07:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 05:07:22.025701 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskEQLg3e/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/postgres -l logfile start
2026-09-27 05:07:24.849 UTC [11287] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 05:07:24.850 UTC [11287] LOG: listening on IPv4 address "127.0.97.20", port 46535
2026-09-27 05:07:24.854 UTC [11287] LOG: listening on Unix socket "/tmp/.s.PGSQL.46535"
2026-09-27 05:07:24.880 UTC [11290] LOG: database system was shut down at 2026-09-27 05:07:23 UTC
2026-09-27 05:07:24.884 UTC [11287] LOG: database system is ready to accept connections
I20260927 05:07:25.888695 388 mini_postgres.cc:96] Postgres bound to 46535
2026-09-27 05:07:25.906 UTC [11298] FATAL: database "slave" does not exist
127.0.97.20:46535 - accepting connections
I20260927 05:07:25.907579 388 mini_ranger.cc:161] Starting Ranger...
I20260927 05:07:25.934973 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 05:07:26.509222 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 05:07:26.509370 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-admin
2026-09-27 05:07:26,820 [I] DB FLAVOR :POSTGRES
2026-09-27 05:07:26,821 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:07:26,821 [I] Checking connection..
2026-09-27 05:07:26,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 05:07:27,093 [I] Checking connection passed.
2026-09-27 05:07:27,094 [I] --------- Verifying version history table ---------
2026-09-27 05:07:27,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:07:27,390 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:07:27,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:07:27,644 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:07:27,644 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 05:07:27,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 05:07:27.910 UTC [11362] WARNING: there is no transaction in progress
2026-09-27 05:07:27,916 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 05:07:27,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:07:28,162 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 05:07:28,163 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:07:28,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 05:07:28,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 05:07:28,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hgs0.c.gcp-upstream.internal','N') ;"
2026-09-27 05:07:29,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:07:29,297 [I] Table x_portal_user does not exist in database ranger
2026-09-27 05:07:29,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:07:29,553 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 05:07:29,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:07:29,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:07:30,094 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 05:07:30,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 05:07:31.219 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.239 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.259 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.275 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.293 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.314 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.373 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.380 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.396 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.407 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.419 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.434 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.444 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.451 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.460 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:31.467 UTC [11470] WARNING: there is no transaction in progress
2026-09-27 05:07:32,196 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 05:07:32,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 05:07:32,423 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 05:07:32,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:07:32,680 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 05:07:32,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:07:32,935 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 05:07:32,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:07:33,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:07:33,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hgs0.c.gcp-upstream.internal';"
2026-09-27 05:07:33,651 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 05:07:33,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 05:07:33,876 [I] DB_PATCHES have already been applied
I20260927 05:07:33.882139 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskEQLg3e/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 05:07:33.882251 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 05:07:33.897037 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:52619
Sep 27, 2026 5:07:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:07:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:07:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 5:07:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 5:07:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 5:07:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 5:07:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 05:07:35.023862 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.116s user 0.000s sys 0.040s
Sep 27, 2026 5:07:56 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 5:07:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 27, 2026 5:07:56 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 5:07:56 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 5:07:57 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 05:07:58.322402 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 05:07:58 dist-test-slave-hgs0 krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485678, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 05:07:58.506207 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 05:07:58.531662 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 05:07:59.572032 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 05:07:59.572209 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-kms
I20260927 05:07:59.572365 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:46535
2026-09-27 05:07:59,995 [I] DB FLAVOR :POSTGRES
2026-09-27 05:07:59,996 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:07:59,996 [I] Checking connection
2026-09-27 05:07:59,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 05:08:00,230 [I] connection success
2026-09-27 05:08:00,230 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 05:08:00,230 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 05:08:00,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 05:08:00,476 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 05:08:00,476 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:08:00,476 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 05:08:00,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46535/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 05:08:00,741 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 05:08:01.265583 388 mini_ranger_kms.cc:310] Created kms service
I20260927 05:08:01.463073 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 05:08:01.510224 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 05:08:01 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485678, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:08:01.913669 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 05:08:01.914007 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 05:08:01.914091 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 05:08:01.931023 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11651
I20260927 05:08:01.931212 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:59627
Sep 27, 2026 5:08:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:08:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:08:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 5:08:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 5:08:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 5:08:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 05:08:04.158484 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.652s user 0.116s sys 0.323s
I20260927 05:08:04.158728 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 05:08:04.158934 388 mini_ranger_kms.cc:401] 127.0.97.20:59627/kms/v1/keys
Sep 27 05:08:04 dist-test-slave-hgs0 krb5kdc[11257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790485684, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 05:08:04 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485684, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 05:08:04 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485684, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 05:08:07 dist-test-slave-hgs0 krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:08:07.397181 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39005
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39005
--ranger_config_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59627/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:08:07.770743 11701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:08:07.771080 11701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:08:07.771241 11701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:08:07.779103 11701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:08:07.779234 11701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:08:07.779318 11701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:08:07.779413 11701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:08:07.779520 11701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:08:07.790643 11701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:59627/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39005
--ranger_config_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:08:07.792119 11701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:08:07.793603 11701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:08:07.808159 11706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:08:07.808228 11707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:08:07.808203 11709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:08:07.808634 11701 server_base.cc:1034] running on GCE node
I20260927 05:08:07.809685 11701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:08:07.811235 11701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:08:07.812489 11701 hybrid_clock.cc:648] HybridClock initialized: now 1790485687812449 us; error 38 us; skew 500 ppm
I20260927 05:08:07.812866 11701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:08:07 dist-test-slave-hgs0 krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:08:07.816267 11701 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:08:07.818004 11701 webserver.cc:460] Webserver started at http://127.0.97.62:42499/ using document root <none> and password file <none>
I20260927 05:08:07.819110 11701 fs_manager.cc:359] Metadata directory not provided
I20260927 05:08:07.819249 11701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:08:07.819576 11701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:08:07 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:08:07.872673 11701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "6498a2d6314643c2b0a89e45a4af9ecc"
format_stamp: "Formatted at 2026-09-27 05:08:07 on dist-test-slave-hgs0"
server_key: "423e234556992b9472bbac842bc8feb3"
server_key_iv: "a4d60d8342282c3e62802fc7973c5931"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:07.873658 11701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "6498a2d6314643c2b0a89e45a4af9ecc"
format_stamp: "Formatted at 2026-09-27 05:08:07 on dist-test-slave-hgs0"
server_key: "423e234556992b9472bbac842bc8feb3"
server_key_iv: "a4d60d8342282c3e62802fc7973c5931"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:07.880285 11701 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 05:08:07.898694 11716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:08:07.900556 11701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:08:07.900779 11701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "6498a2d6314643c2b0a89e45a4af9ecc"
format_stamp: "Formatted at 2026-09-27 05:08:07 on dist-test-slave-hgs0"
server_key: "423e234556992b9472bbac842bc8feb3"
server_key_iv: "a4d60d8342282c3e62802fc7973c5931"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:07.901054 11701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:08:07.940973 11701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:08:07.945574 11701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:08:07.945969 11701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:08:07.967980 11701 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39005
I20260927 05:08:07.968031 11767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39005 every 8 connection(s)
I20260927 05:08:07.969545 11701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:08:07.973801 11768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:08:07.979728 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 11701
I20260927 05:08:07.980028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:08:07.980899 388 external_mini_cluster.cc:1428] Setting key 6814096f7cb301be589186ae01e2d499
I20260927 05:08:07.988847 11768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc: Bootstrap starting.
I20260927 05:08:07.992383 11768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc: Neither blocks nor log segments found. Creating new log.
I20260927 05:08:07.993543 11768 log.cc:824] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc: Log is configured to *not* fsync() on all Append() calls
I20260927 05:08:07.996780 11768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc: No bootstrap required, opened a new log
Sep 27 05:08:07 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:08:08.007045 11768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6498a2d6314643c2b0a89e45a4af9ecc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39005 } }
I20260927 05:08:08.007447 11768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:08:08.007586 11768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6498a2d6314643c2b0a89e45a4af9ecc, State: Initialized, Role: FOLLOWER
I20260927 05:08:08.008364 11768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6498a2d6314643c2b0a89e45a4af9ecc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39005 } }
I20260927 05:08:08.008665 11768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:08:08.008860 11768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:08:08.009130 11768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:08:08.010452 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:07.983835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.0.97.62:39005)
0927 05:08:07.986987 (+ 3152us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:07.987007 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:07.987494 (+ 487us) server_negotiation.cc:407] Connection header received
0927 05:08:07.989957 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:07.989983 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:07.990447 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:07.990907 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:08:07.992306 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:07.993801 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:07.995100 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:07.996347 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:07.996551 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:08.000847 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:08:08.000858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:08:08.000881 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:08:08.000930 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:08:08.005067 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:08.006033 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:08.006043 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:08.006054 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:08.006253 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:08.006825 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:08.006831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:08.006835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:08.007707 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:08:08.008092 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:08.008665 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 05:08:08.009163 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2864,"thread_start_us":2546,"threads_started":1}
I20260927 05:08:08.011909 11768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6498a2d6314643c2b0a89e45a4af9ecc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39005 } }
I20260927 05:08:08.012609 11768 leader_election.cc:302] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6498a2d6314643c2b0a89e45a4af9ecc; no voters:
I20260927 05:08:08.013859 11768 leader_election.cc:288] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:08:08.014060 11773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:08:08.015363 11773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [term 1 LEADER]: Becoming Leader. State: Replica: 6498a2d6314643c2b0a89e45a4af9ecc, State: Running, Role: LEADER
I20260927 05:08:08.016055 11773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6498a2d6314643c2b0a89e45a4af9ecc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39005 } }
I20260927 05:08:08.016404 11768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:08:08.019387 11775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6498a2d6314643c2b0a89e45a4af9ecc. Latest consensus state: current_term: 1 leader_uuid: "6498a2d6314643c2b0a89e45a4af9ecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6498a2d6314643c2b0a89e45a4af9ecc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39005 } } }
I20260927 05:08:08.019421 11774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6498a2d6314643c2b0a89e45a4af9ecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6498a2d6314643c2b0a89e45a4af9ecc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39005 } } }
I20260927 05:08:08.020295 11775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [sys.catalog]: This master's current role is: LEADER
I20260927 05:08:08.020449 11774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc [sys.catalog]: This master's current role is: LEADER
I20260927 05:08:08.021378 11782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:08:08.028156 11782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:08:08.028790 11768 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 05:08:08.047694 11782 catalog_manager.cc:1347] Generated new cluster ID: da3f883e67474615b23ba0fc16be1ca0
I20260927 05:08:08.047848 11782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:08:08.067124 11782 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:08:08.069350 11782 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:08:08.079595 11782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc: Generated new TSK 0
I20260927 05:08:08.080983 11782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:08:08.932662 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39005
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:08:09 dist-test-slave-hgs0 krb5kdc[11257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790485689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 05:08:09.395236 11813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:08:09.395598 11813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:08:09.395767 11813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:08:09.403517 11813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:08:09.403685 11813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:08:09.403867 11813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:08:09.416204 11813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:59627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:08:09.417644 11813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:08:09.419214 11813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:08:09.443454 11824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:08:09.445369 11823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:08:09.450007 11813 server_base.cc:1034] running on GCE node
W20260927 05:08:09.449819 11826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:08:09.450960 11813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:08:09.452229 11813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:08:09.453522 11813 hybrid_clock.cc:648] HybridClock initialized: now 1790485689453441 us; error 73 us; skew 500 ppm
I20260927 05:08:09.454000 11813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:08:09 dist-test-slave-hgs0 krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:08:09.457928 11813 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:08:09.460935 11813 webserver.cc:460] Webserver started at http://127.0.97.1:32897/ using document root <none> and password file <none>
I20260927 05:08:09.462056 11813 fs_manager.cc:359] Metadata directory not provided
I20260927 05:08:09.462229 11813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:08:09.462616 11813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:08:09 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:08:09.482990 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "49094e537b334f7eb723a36e03b84cbb"
format_stamp: "Formatted at 2026-09-27 05:08:09 on dist-test-slave-hgs0"
server_key: "1cd4bf46ad2bc164e6aa6667928f78b3"
server_key_iv: "afa1539418a6f2922a0798bee1e61b50"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:09.487888 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "49094e537b334f7eb723a36e03b84cbb"
format_stamp: "Formatted at 2026-09-27 05:08:09 on dist-test-slave-hgs0"
server_key: "1cd4bf46ad2bc164e6aa6667928f78b3"
server_key_iv: "afa1539418a6f2922a0798bee1e61b50"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:09.496791 11813 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s
I20260927 05:08:09.529039 11833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:08:09.531356 11813 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.008s sys 0.004s
I20260927 05:08:09.531611 11813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "49094e537b334f7eb723a36e03b84cbb"
format_stamp: "Formatted at 2026-09-27 05:08:09 on dist-test-slave-hgs0"
server_key: "1cd4bf46ad2bc164e6aa6667928f78b3"
server_key_iv: "afa1539418a6f2922a0798bee1e61b50"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:09.531919 11813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:08:09.569372 11813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:08:09.575389 11813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:08:09.575847 11813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:08:09.577350 11813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:08:09.580219 11813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:08:09.580358 11813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:08:09.580505 11813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:08:09.580582 11813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:08:09.641072 11813 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42179
I20260927 05:08:09.641083 11945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42179 every 8 connection(s)
I20260927 05:08:09.642688 11813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:08:09.644217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 11813
I20260927 05:08:09.644484 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:08:09.645491 388 external_mini_cluster.cc:1428] Setting key 36fe956c8701eb4ecc804c4db8a55299
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 27 05:08:09 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:08:09.695577 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:09.647909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56083 (local address 127.0.97.62:39005)
0927 05:08:09.663208 (+ 15299us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:09.663217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:09.667882 (+ 4665us) server_negotiation.cc:407] Connection header received
0927 05:08:09.668003 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:09.668012 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:09.668124 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:09.668240 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:08:09.672034 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:09.673482 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:09.676746 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:09.677545 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:09.677668 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:09.686031 (+ 8363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:08:09.686043 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:08:09.686052 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:08:09.686113 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:08:09.690683 (+ 4570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:09.692034 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:09.692044 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:09.692050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:09.692233 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:09.692883 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:09.692891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:09.692909 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:09.693341 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:08:09.693506 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:09.694731 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0927 05:08:09.695043 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15121,"thread_start_us":19768,"threads_started":1}
I20260927 05:08:09.698556 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:09.648831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39005 (local address 127.0.97.1:56083)
0927 05:08:09.663905 (+ 15074us) negotiation.cc:107] Waiting for socket to connect
0927 05:08:09.663997 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0927 05:08:09.667218 (+ 3221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:08:09.669878 (+ 2660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:08:09.669911 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:08:09.670680 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:08:09.671688 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:09.671723 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:09.673762 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:09.673771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:09.676486 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:09.676513 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:09.678612 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:09.678623 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:09.679022 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:09.679996 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:08:09.680049 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:08:09.685612 (+ 5563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:08:09.691001 (+ 5389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:08:09.691015 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:08:09.691039 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:08:09.691804 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:08:09.692417 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:08:09.692425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:08:09.692432 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:08:09.692738 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:08:09.693628 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:08:09.693639 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:08:09.694203 (+ 564us) client_negotiation.cc:769] Sending connection context
0927 05:08:09.696084 (+ 1881us) client_negotiation.cc:240] Negotiation successful
0927 05:08:09.696776 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14636,"thread_start_us":17936,"threads_started":1}
I20260927 05:08:09.701006 11946 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39005
I20260927 05:08:09.701435 11946 heartbeater.cc:464] Registering TS with master...
I20260927 05:08:09.702488 11946 heartbeater.cc:511] Master 127.0.97.62:39005 requested a full tablet report, sending...
I20260927 05:08:09.705726 11733 ts_manager.cc:194] Registered new tserver with Master: 49094e537b334f7eb723a36e03b84cbb (127.0.97.1:42179)
I20260927 05:08:09.708354 11733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:08:09.723851 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39005
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:08:09 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 05:08:10 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 05:08:10.226217 11952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:08:10.226766 11952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:08:10.226974 11952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:08:10.240487 11952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:08:10.240756 11952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:08:10.240993 11952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:08:10.259307 11952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:59627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:08:10.261612 11952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:08:10.263574 11952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:08:10.313455 11960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:08:10.318537 11959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:08:10.321543 11963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:08:10.322444 11952 server_base.cc:1034] running on GCE node
I20260927 05:08:10.323714 11952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:08:10.332309 11952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:08:10.335866 11952 hybrid_clock.cc:648] HybridClock initialized: now 1790485690335775 us; error 92 us; skew 500 ppm
I20260927 05:08:10.336372 11952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:08:10 dist-test-slave-hgs0 krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:08:10.347577 11952 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:08:10.358565 11952 webserver.cc:460] Webserver started at http://127.0.97.2:40423/ using document root <none> and password file <none>
I20260927 05:08:10.359786 11952 fs_manager.cc:359] Metadata directory not provided
I20260927 05:08:10.359992 11952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:08:10.360419 11952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:08:10 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:08:10.380223 11952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "92387153f47b41b4926e2bb59a61fecb"
format_stamp: "Formatted at 2026-09-27 05:08:10 on dist-test-slave-hgs0"
server_key: "268e68eb37689d7f25dfa9204b9aab82"
server_key_iv: "52ee82ed558c53fa2cf30191c4c6a248"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:10.381672 11952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "92387153f47b41b4926e2bb59a61fecb"
format_stamp: "Formatted at 2026-09-27 05:08:10 on dist-test-slave-hgs0"
server_key: "268e68eb37689d7f25dfa9204b9aab82"
server_key_iv: "52ee82ed558c53fa2cf30191c4c6a248"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:10.418596 11952 fs_manager.cc:691] Time spent creating directory manager: real 0.036s user 0.018s sys 0.016s
I20260927 05:08:10.456257 11969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:08:10.458585 11952 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.006s sys 0.005s
I20260927 05:08:10.458914 11952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "92387153f47b41b4926e2bb59a61fecb"
format_stamp: "Formatted at 2026-09-27 05:08:10 on dist-test-slave-hgs0"
server_key: "268e68eb37689d7f25dfa9204b9aab82"
server_key_iv: "52ee82ed558c53fa2cf30191c4c6a248"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:10.459257 11952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:08:10.523011 11952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:08:10.528681 11952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:08:10.529088 11952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:08:10.530407 11952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:08:10.532428 11952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:08:10.532567 11952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 05:08:10.532724 11952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:08:10.532820 11952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:08:10.583184 11952 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42619
I20260927 05:08:10.583211 12088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42619 every 8 connection(s)
I20260927 05:08:10.585152 11952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:08:10.590701 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 11952
I20260927 05:08:10.590888 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:08:10.591367 388 external_mini_cluster.cc:1428] Setting key 0ca442c11d42b7550ff5830a61b081a8
Sep 27 05:08:10 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:08:10.624955 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:10.589164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39005 (local address 127.0.97.2:47667)
0927 05:08:10.596639 (+ 7475us) negotiation.cc:107] Waiting for socket to connect
0927 05:08:10.596709 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 05:08:10.599564 (+ 2855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:08:10.601232 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:08:10.601247 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:08:10.602014 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:08:10.602952 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:10.602983 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:10.604668 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:10.604675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:10.607101 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:10.607123 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:10.608326 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:10.608334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:10.608682 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:10.612282 (+ 3600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:08:10.612321 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:08:10.615424 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:08:10.619837 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:08:10.619848 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:08:10.619872 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:08:10.620551 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:08:10.621043 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:08:10.621048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:08:10.621052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:08:10.621304 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:08:10.622026 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:08:10.622035 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:08:10.622513 (+ 478us) client_negotiation.cc:769] Sending connection context
0927 05:08:10.622841 (+ 328us) client_negotiation.cc:240] Negotiation successful
0927 05:08:10.623418 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7127,"thread_start_us":6900,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:08:10.626103 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:10.588205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47667 (local address 127.0.97.62:39005)
0927 05:08:10.600426 (+ 12221us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:10.600433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:10.600454 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:08:10.600519 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:10.600524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:10.600619 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:10.600735 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:08:10.603206 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:10.604484 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:10.607319 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:10.608048 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:10.608211 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:10.615781 (+ 7570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:08:10.615801 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:08:10.615806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:08:10.615867 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:08:10.619556 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:10.620729 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:10.620736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:10.620739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:10.620907 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:10.621436 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:10.621440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:10.621444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:10.621812 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:08:10.621941 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:10.625504 (+ 3563us) server_negotiation.cc:299] Negotiation successful
0927 05:08:10.625745 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12067,"thread_start_us":11944,"threads_started":1}
I20260927 05:08:10.628127 12089 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39005
I20260927 05:08:10.628511 12089 heartbeater.cc:464] Registering TS with master...
I20260927 05:08:10.629549 12089 heartbeater.cc:511] Master 127.0.97.62:39005 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:08:10.633992 11733 ts_manager.cc:194] Registered new tserver with Master: 92387153f47b41b4926e2bb59a61fecb (127.0.97.2:42619)
I20260927 05:08:10.635138 11733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:08:10.669164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39005
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:08:10.713778 11946 heartbeater.cc:503] Master 127.0.97.62:39005 was elected leader, sending a full tablet report...
W20260927 05:08:11.102202 12095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:08:11.102523 12095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:08:11.102681 12095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:08:11.111320 12095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:08:11.111452 12095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:08:11.111675 12095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:08:11.122044 12095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:59627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:08:11.123363 12095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:08:11.124851 12095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:08:11.139209 12100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:08:11.139245 12101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:08:11.139248 12103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:08:11.140947 12095 server_base.cc:1034] running on GCE node
I20260927 05:08:11.141634 12095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:08:11.142658 12095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:08:11.143908 12095 hybrid_clock.cc:648] HybridClock initialized: now 1790485691143848 us; error 53 us; skew 500 ppm
I20260927 05:08:11.144316 12095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:08:11 dist-test-slave-hgs0 krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:08:11.147872 12095 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:08:11.149588 12095 webserver.cc:460] Webserver started at http://127.0.97.3:44987/ using document root <none> and password file <none>
I20260927 05:08:11.150444 12095 fs_manager.cc:359] Metadata directory not provided
I20260927 05:08:11.150561 12095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:08:11.150848 12095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:08:11 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:08:11.168877 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "36a7694b32834289ba7cea58cfb07011"
format_stamp: "Formatted at 2026-09-27 05:08:11 on dist-test-slave-hgs0"
server_key: "dc71fd774b00ee09fe67f67f72b2d464"
server_key_iv: "ba3d5c5f4994310477d2b2dc645f1cb3"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:11.169818 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "36a7694b32834289ba7cea58cfb07011"
format_stamp: "Formatted at 2026-09-27 05:08:11 on dist-test-slave-hgs0"
server_key: "dc71fd774b00ee09fe67f67f72b2d464"
server_key_iv: "ba3d5c5f4994310477d2b2dc645f1cb3"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:11.175714 12095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:08:11.193651 12110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:08:11.195544 12095 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:08:11.195753 12095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "36a7694b32834289ba7cea58cfb07011"
format_stamp: "Formatted at 2026-09-27 05:08:11 on dist-test-slave-hgs0"
server_key: "dc71fd774b00ee09fe67f67f72b2d464"
server_key_iv: "ba3d5c5f4994310477d2b2dc645f1cb3"
server_key_version: "kuduclusterkey@0"
I20260927 05:08:11.196048 12095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:08:11.242676 12095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:08:11.246906 12095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:08:11.247171 12095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:08:11.248550 12095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:08:11.250013 12095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:08:11.250088 12095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:08:11.250171 12095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:08:11.250214 12095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:08:11.287662 12095 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34063
I20260927 05:08:11.287695 12222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34063 every 8 connection(s)
I20260927 05:08:11.289026 12095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:08:11.293241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 12095
I20260927 05:08:11.293458 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:08:11.294013 388 external_mini_cluster.cc:1428] Setting key f65bd75d612ac423d44ddc555898fe4e
Sep 27 05:08:11 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:08:11.312230 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.291993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43811 (local address 127.0.97.62:39005)
0927 05:08:11.292516 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.292523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.293906 (+ 1383us) server_negotiation.cc:407] Connection header received
0927 05:08:11.296859 (+ 2953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.296866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.296955 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.297079 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:08:11.298926 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.299758 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.301535 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.302026 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.302135 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.305786 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:08:11.305791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:08:11.305805 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:08:11.305839 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:08:11.308845 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:11.309695 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:11.309705 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:11.309712 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:11.309884 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:11.310434 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:11.310439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:11.310443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:11.310804 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:08:11.310924 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.311691 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.311885 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":245,"threads_started":1}
I20260927 05:08:11.313249 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.292751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39005 (local address 127.0.97.3:43811)
0927 05:08:11.293659 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 05:08:11.293709 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:08:11.296513 (+ 2804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:08:11.297408 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:08:11.297419 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:08:11.298130 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:08:11.298755 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.298779 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.299921 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.299925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.301367 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.301379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.302257 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.302263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.302573 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.303319 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:08:11.303342 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:08:11.305558 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:08:11.309011 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:08:11.309019 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:08:11.309035 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:08:11.309575 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:08:11.310017 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:08:11.310020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:08:11.310022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:08:11.310218 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:08:11.310935 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:08:11.310941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:08:11.311301 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 05:08:11.311558 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 05:08:11.312053 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":354,"threads_started":1}
I20260927 05:08:11.314728 12223 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39005
I20260927 05:08:11.315058 12223 heartbeater.cc:464] Registering TS with master...
I20260927 05:08:11.315807 12223 heartbeater.cc:511] Master 127.0.97.62:39005 requested a full tablet report, sending...
I20260927 05:08:11.317346 11733 ts_manager.cc:194] Registered new tserver with Master: 36a7694b32834289ba7cea58cfb07011 (127.0.97.3:34063)
I20260927 05:08:11.318189 11733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43811
I20260927 05:08:11.323069 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:08:11.341161 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.328615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54524 (local address 127.0.97.62:39005)
0927 05:08:11.328866 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.328877 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.329245 (+ 368us) server_negotiation.cc:407] Connection header received
0927 05:08:11.329398 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.329402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.329481 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.329570 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:08:11.330932 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.331709 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.333085 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.333558 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.333662 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.335867 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:08:11.335871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:08:11.335874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:08:11.335904 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:08:11.338628 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:11.339397 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:11.339405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:11.339414 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:11.339531 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:11.339879 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:11.339882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:11.339884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:11.340195 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:08:11.340379 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.340674 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.340846 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:08:11.345386 11733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:08:11.352897 11733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:08:11.377162 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.365945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.0.97.2:42619)
0927 05:08:11.366874 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.366887 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.367864 (+ 977us) server_negotiation.cc:407] Connection header received
0927 05:08:11.368118 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.368127 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.368389 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.368591 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:08:11.369749 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.371189 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.374041 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.375507 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.375647 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.375873 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.376659 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.376838 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":691,"threads_started":1}
I20260927 05:08:11.377207 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.365769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42619 (local address 127.0.0.1:56730)
0927 05:08:11.367581 (+ 1812us) negotiation.cc:107] Waiting for socket to connect
0927 05:08:11.367639 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 05:08:11.367844 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:08:11.368837 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:08:11.368843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:08:11.368874 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:08:11.369344 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.369360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.371320 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.371327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.373862 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.373877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.376314 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.376323 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.376464 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.376487 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:08:11.376604 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 05:08:11.376816 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1564,"mutex_wait_us":1,"thread_start_us":1948,"threads_started":1}
I20260927 05:08:11.378155 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.366678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34063 (local address 127.0.0.1:40892)
0927 05:08:11.367853 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0927 05:08:11.367870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:08:11.368009 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:08:11.370299 (+ 2290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:08:11.370309 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:08:11.370339 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:08:11.370769 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.370783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.372460 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.372465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.374382 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.374392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.376169 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.376176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.376385 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.376450 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 05:08:11.376581 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 05:08:11.377727 (+ 1146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":46,"thread_start_us":470,"threads_started":1}
I20260927 05:08:11.378182 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.367415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40892 (local address 127.0.97.3:34063)
0927 05:08:11.369458 (+ 2043us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.369468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.369502 (+ 34us) server_negotiation.cc:407] Connection header received
0927 05:08:11.369599 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.369606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.369854 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.370036 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:08:11.370942 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.372309 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.374533 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.375963 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.376096 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.376328 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.377668 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.377854 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":1305,"threads_started":1}
I20260927 05:08:11.378741 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.368567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.0.97.1:42179)
0927 05:08:11.369318 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.369328 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.370661 (+ 1333us) server_negotiation.cc:407] Connection header received
0927 05:08:11.370905 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.370913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.371133 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.371318 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:08:11.372040 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.373487 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.375872 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.377370 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.377496 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.377698 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.378227 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.378400 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":940,"threads_started":1}
I20260927 05:08:11.379591 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.368166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42179 (local address 127.0.0.1:41702)
0927 05:08:11.370539 (+ 2373us) negotiation.cc:107] Waiting for socket to connect
0927 05:08:11.370556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:08:11.370709 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:08:11.371389 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:08:11.371392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:08:11.371427 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:08:11.371846 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.371856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.373665 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.373670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.375685 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.375697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.377865 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.377872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.378045 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.378070 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 05:08:11.378183 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 05:08:11.378305 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2250,"thread_start_us":2088,"threads_started":1}
I20260927 05:08:11.381901 12024 tablet_service.cc:1467] Processing CreateTablet for tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef (DEFAULT_TABLE table=test-table [id=57f3e76100af42bc92d58bcd46a640df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:08:11.381992 12158 tablet_service.cc:1467] Processing CreateTablet for tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef (DEFAULT_TABLE table=test-table [id=57f3e76100af42bc92d58bcd46a640df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:08:11.383188 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:08:11.382910 11881 tablet_service.cc:1467] Processing CreateTablet for tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef (DEFAULT_TABLE table=test-table [id=57f3e76100af42bc92d58bcd46a640df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:08:11.383414 12158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:08:11.384274 11881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:08:11.394635 12242 tablet_bootstrap.cc:492] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: Bootstrap starting.
I20260927 05:08:11.399056 12242 tablet_bootstrap.cc:654] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: Neither blocks nor log segments found. Creating new log.
I20260927 05:08:11.400277 12242 log.cc:824] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: Log is configured to *not* fsync() on all Append() calls
I20260927 05:08:11.400705 12243 tablet_bootstrap.cc:492] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: Bootstrap starting.
I20260927 05:08:11.401859 12244 tablet_bootstrap.cc:492] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: Bootstrap starting.
I20260927 05:08:11.405479 12244 tablet_bootstrap.cc:654] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: Neither blocks nor log segments found. Creating new log.
I20260927 05:08:11.405594 12243 tablet_bootstrap.cc:654] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: Neither blocks nor log segments found. Creating new log.
I20260927 05:08:11.406070 12242 tablet_bootstrap.cc:492] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: No bootstrap required, opened a new log
I20260927 05:08:11.406414 12242 ts_tablet_manager.cc:1397] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260927 05:08:11.406940 12244 log.cc:824] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: Log is configured to *not* fsync() on all Append() calls
I20260927 05:08:11.407496 12243 log.cc:824] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: Log is configured to *not* fsync() on all Append() calls
I20260927 05:08:11.409916 12244 tablet_bootstrap.cc:492] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: No bootstrap required, opened a new log
I20260927 05:08:11.410192 12244 ts_tablet_manager.cc:1397] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 05:08:11.415367 12243 tablet_bootstrap.cc:492] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: No bootstrap required, opened a new log
I20260927 05:08:11.415676 12243 ts_tablet_manager.cc:1397] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260927 05:08:11.418316 12242 raft_consensus.cc:348] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } } peers { permanent_uuid: "49094e537b334f7eb723a36e03b84cbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42179 } } peers { permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 } }
I20260927 05:08:11.418831 12242 raft_consensus.cc:374] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:08:11.418982 12242 raft_consensus.cc:729] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92387153f47b41b4926e2bb59a61fecb, State: Initialized, Role: FOLLOWER
I20260927 05:08:11.418783 12244 raft_consensus.cc:348] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } } peers { permanent_uuid: "49094e537b334f7eb723a36e03b84cbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42179 } } peers { permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 } }
I20260927 05:08:11.419250 12244 raft_consensus.cc:374] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:08:11.419423 12244 raft_consensus.cc:729] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49094e537b334f7eb723a36e03b84cbb, State: Initialized, Role: FOLLOWER
I20260927 05:08:11.419669 12242 consensus_queue.cc:260] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } } peers { permanent_uuid: "49094e537b334f7eb723a36e03b84cbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42179 } } peers { permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 } }
I20260927 05:08:11.420096 12244 consensus_queue.cc:260] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } } peers { permanent_uuid: "49094e537b334f7eb723a36e03b84cbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42179 } } peers { permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 } }
I20260927 05:08:11.421550 12089 heartbeater.cc:503] Master 127.0.97.62:39005 was elected leader, sending a full tablet report...
I20260927 05:08:11.422204 12242 ts_tablet_manager.cc:1428] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260927 05:08:11.424415 12244 ts_tablet_manager.cc:1428] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 05:08:11.428121 12243 raft_consensus.cc:348] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } } peers { permanent_uuid: "49094e537b334f7eb723a36e03b84cbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42179 } } peers { permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 } }
I20260927 05:08:11.428718 12243 raft_consensus.cc:374] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:08:11.428900 12243 raft_consensus.cc:729] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a7694b32834289ba7cea58cfb07011, State: Initialized, Role: FOLLOWER
I20260927 05:08:11.429620 12243 consensus_queue.cc:260] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } } peers { permanent_uuid: "49094e537b334f7eb723a36e03b84cbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42179 } } peers { permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 } }
I20260927 05:08:11.429899 12249 raft_consensus.cc:482] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:08:11.430238 12249 raft_consensus.cc:504] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } } peers { permanent_uuid: "49094e537b334f7eb723a36e03b84cbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42179 } } peers { permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 } }
I20260927 05:08:11.430989 12223 heartbeater.cc:503] Master 127.0.97.62:39005 was elected leader, sending a full tablet report...
I20260927 05:08:11.431905 12243 ts_tablet_manager.cc:1428] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260927 05:08:11.433373 12249 leader_election.cc:288] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92387153f47b41b4926e2bb59a61fecb (127.0.97.2:42619), 36a7694b32834289ba7cea58cfb07011 (127.0.97.3:34063)
I20260927 05:08:11.441344 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.434209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36835 (local address 127.0.97.2:42619)
0927 05:08:11.434404 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.434412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.434434 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:08:11.434707 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.434714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.434801 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.434874 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:08:11.435674 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.437154 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.439491 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.440374 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.440474 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.440545 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.440953 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.441082 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:08:11.441474 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.433639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42619 (local address 127.0.97.1:36835)
0927 05:08:11.434356 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 05:08:11.434378 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:08:11.434519 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:08:11.435079 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:08:11.435084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:08:11.435111 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:08:11.435519 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.435539 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.437292 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.437298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.439335 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.439349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.440590 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.440597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.440779 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.440815 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:08:11.440926 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 05:08:11.441059 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":1,"thread_start_us":794,"threads_started":1}
I20260927 05:08:11.442726 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.433851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34063 (local address 127.0.97.1:51203)
0927 05:08:11.436907 (+ 3056us) negotiation.cc:107] Waiting for socket to connect
0927 05:08:11.436930 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:08:11.437049 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:08:11.437350 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:08:11.437355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:08:11.437400 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:08:11.437845 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.437856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.438909 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.438915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.440663 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:08:11.440674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.442057 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.442067 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:08:11.442272 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.442307 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:08:11.442411 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:08:11.442500 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2908,"thread_start_us":2754,"threads_started":1}
I20260927 05:08:11.442833 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.435205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51203 (local address 127.0.97.3:34063)
0927 05:08:11.435431 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.435440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.437026 (+ 1586us) server_negotiation.cc:407] Connection header received
0927 05:08:11.437166 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.437174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.437243 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.437329 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:08:11.438044 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.438757 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.440821 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.441740 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.441879 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.441988 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.442463 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.442600 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:08:11.442924 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dded69cb1c4b3f9f4ec4f98dabc2ef" candidate_uuid: "49094e537b334f7eb723a36e03b84cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92387153f47b41b4926e2bb59a61fecb" is_pre_election: true
I20260927 05:08:11.443514 12044 raft_consensus.cc:2454] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49094e537b334f7eb723a36e03b84cbb in term 0.
I20260927 05:08:11.443737 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dded69cb1c4b3f9f4ec4f98dabc2ef" candidate_uuid: "49094e537b334f7eb723a36e03b84cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a7694b32834289ba7cea58cfb07011" is_pre_election: true
I20260927 05:08:11.444307 12178 raft_consensus.cc:2454] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49094e537b334f7eb723a36e03b84cbb in term 0.
I20260927 05:08:11.444367 11834 leader_election.cc:302] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49094e537b334f7eb723a36e03b84cbb, 92387153f47b41b4926e2bb59a61fecb; no voters:
I20260927 05:08:11.444844 12249 raft_consensus.cc:2781] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:08:11.445006 12249 raft_consensus.cc:482] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:08:11.445119 12249 raft_consensus.cc:3037] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:08:11.448016 12249 raft_consensus.cc:504] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } } peers { permanent_uuid: "49094e537b334f7eb723a36e03b84cbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42179 } } peers { permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 } }
I20260927 05:08:11.448638 12249 leader_election.cc:288] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [CANDIDATE]: Term 1 election: Requested vote from peers 92387153f47b41b4926e2bb59a61fecb (127.0.97.2:42619), 36a7694b32834289ba7cea58cfb07011 (127.0.97.3:34063)
I20260927 05:08:11.449127 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dded69cb1c4b3f9f4ec4f98dabc2ef" candidate_uuid: "49094e537b334f7eb723a36e03b84cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92387153f47b41b4926e2bb59a61fecb"
I20260927 05:08:11.449127 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2dded69cb1c4b3f9f4ec4f98dabc2ef" candidate_uuid: "49094e537b334f7eb723a36e03b84cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a7694b32834289ba7cea58cfb07011"
I20260927 05:08:11.449339 12044 raft_consensus.cc:3037] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:08:11.449389 12178 raft_consensus.cc:3037] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:08:11.452685 12178 raft_consensus.cc:2454] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49094e537b334f7eb723a36e03b84cbb in term 1.
I20260927 05:08:11.452685 12044 raft_consensus.cc:2454] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49094e537b334f7eb723a36e03b84cbb in term 1.
I20260927 05:08:11.453181 11834 leader_election.cc:302] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49094e537b334f7eb723a36e03b84cbb, 92387153f47b41b4926e2bb59a61fecb; no voters:
I20260927 05:08:11.453567 12249 raft_consensus.cc:2781] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:08:11.453981 12249 raft_consensus.cc:686] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 1 LEADER]: Becoming Leader. State: Replica: 49094e537b334f7eb723a36e03b84cbb, State: Running, Role: LEADER
I20260927 05:08:11.454478 12249 consensus_queue.cc:237] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } } peers { permanent_uuid: "49094e537b334f7eb723a36e03b84cbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42179 } } peers { permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 } }
I20260927 05:08:11.459582 11732 catalog_manager.cc:5495] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 49094e537b334f7eb723a36e03b84cbb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "49094e537b334f7eb723a36e03b84cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49094e537b334f7eb723a36e03b84cbb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 } health_report { overall_health: UNKNOWN } } }
I20260927 05:08:11.481067 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.474793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.0.97.1:42179)
0927 05:08:11.475018 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.475026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.475053 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:08:11.475337 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.475342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.475433 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.475594 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:08:11.476010 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.476736 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.478738 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.479155 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.479295 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.479826 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:08:11.480470 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:08:11.480639 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.480695 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.480781 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:08:11.489696 12044 raft_consensus.cc:1261] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb [term 1 FOLLOWER]: Refusing update from remote peer 49094e537b334f7eb723a36e03b84cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:08:11.489693 12178 raft_consensus.cc:1261] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011 [term 1 FOLLOWER]: Refusing update from remote peer 49094e537b334f7eb723a36e03b84cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:08:11.490833 12253 consensus_queue.cc:1035] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "36a7694b32834289ba7cea58cfb07011" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:08:11.491241 12249 consensus_queue.cc:1035] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92387153f47b41b4926e2bb59a61fecb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:08:11.505191 12257 mvcc.cc:205] Tried to move back new op lower bound from 7333829392330915840 to 7333829392201355264. Current Snapshot: MvccSnapshot[applied={T|T < 7333829392330915840}]
I20260927 05:08:11.507586 12256 mvcc.cc:205] Tried to move back new op lower bound from 7333829392330915840 to 7333829392201355264. Current Snapshot: MvccSnapshot[applied={T|T < 7333829392330915840}]
I20260927 05:08:11.522137 11732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:08:11.522511 11732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:08:11.526401 11732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6498a2d6314643c2b0a89e45a4af9ecc: Sending DeleteTablet for 3 replicas of tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef
I20260927 05:08:11.527737 12024 tablet_service.cc:1514] Processing DeleteTablet for tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:08:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56730
I20260927 05:08:11.527737 12158 tablet_service.cc:1514] Processing DeleteTablet for tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:08:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40892
I20260927 05:08:11.528064 11881 tablet_service.cc:1514] Processing DeleteTablet for tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:08:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41702
I20260927 05:08:11.530035 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 11813
I20260927 05:08:11.540621 12266 tablet_replica.cc:330] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: stopping tablet replica
I20260927 05:08:11.542088 12267 tablet_replica.cc:330] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: stopping tablet replica
I20260927 05:08:11.542533 12267 raft_consensus.cc:2229] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:08:11.543020 12267 raft_consensus.cc:2258] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:08:11.543643 12266 raft_consensus.cc:2229] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:08:11.544205 12266 raft_consensus.cc:2258] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:08:11.545239 12268 tablet_replica.cc:330] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: stopping tablet replica
I20260927 05:08:11.545750 12268 raft_consensus.cc:2229] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:08:11.546558 12268 raft_consensus.cc:2258] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:08:11.547324 12267 ts_tablet_manager.cc:1905] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:08:11.548919 12268 ts_tablet_manager.cc:1905] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:08:11.552276 12266 ts_tablet_manager.cc:1905] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:08:11.555016 12267 ts_tablet_manager.cc:1918] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:08:11.555303 12267 log.cc:1196] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/d2dded69cb1c4b3f9f4ec4f98dabc2ef
I20260927 05:08:11.556018 12267 ts_tablet_manager.cc:1939] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 92387153f47b41b4926e2bb59a61fecb: Deleting consensus metadata
I20260927 05:08:11.557186 12268 ts_tablet_manager.cc:1918] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:08:11.557507 12268 log.cc:1196] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/d2dded69cb1c4b3f9f4ec4f98dabc2ef
I20260927 05:08:11.558202 12268 ts_tablet_manager.cc:1939] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 49094e537b334f7eb723a36e03b84cbb: Deleting consensus metadata
I20260927 05:08:11.558458 11717 catalog_manager.cc:4841] TS 92387153f47b41b4926e2bb59a61fecb (127.0.97.2:42619): tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef (table test-table [id=57f3e76100af42bc92d58bcd46a640df]) successfully deleted
I20260927 05:08:11.560031 11717 catalog_manager.cc:4841] TS 49094e537b334f7eb723a36e03b84cbb (127.0.97.1:42179): tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef (table test-table [id=57f3e76100af42bc92d58bcd46a640df]) successfully deleted
Sep 27 05:08:11 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:08:11.566614 12266 ts_tablet_manager.cc:1918] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:08:11.566808 12266 log.cc:1196] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/d2dded69cb1c4b3f9f4ec4f98dabc2ef
I20260927 05:08:11.567464 12266 ts_tablet_manager.cc:1939] T d2dded69cb1c4b3f9f4ec4f98dabc2ef P 36a7694b32834289ba7cea58cfb07011: Deleting consensus metadata
I20260927 05:08:11.569511 11718 catalog_manager.cc:4841] TS 36a7694b32834289ba7cea58cfb07011 (127.0.97.3:34063): tablet d2dded69cb1c4b3f9f4ec4f98dabc2ef (table test-table [id=57f3e76100af42bc92d58bcd46a640df]) successfully deleted
I20260927 05:08:11.570017 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.532181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.0.97.1:42179)
0927 05:08:11.532449 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.532460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.544548 (+ 12088us) server_negotiation.cc:407] Connection header received
0927 05:08:11.544876 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.544897 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.545005 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.545136 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:08:11.547419 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.548660 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.552575 (+ 3915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.553356 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.553490 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.561989 (+ 8499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:08:11.562012 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:08:11.562034 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:08:11.562091 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:08:11.565994 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:11.566958 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:11.566969 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:11.566973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:11.567210 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:11.567808 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:11.567813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:11.567817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:11.568299 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:08:11.568636 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.568971 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.569224 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:08:11.570770 11941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:08:11.689257 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 11813
I20260927 05:08:11.704792 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 11952
Sep 27 05:08:11 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:08:11.721019 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.705691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.0.97.2:42619)
0927 05:08:11.705883 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.705888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.705907 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:08:11.705966 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.705969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.706044 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.706132 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:08:11.707420 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.708271 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.709592 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.710011 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.710111 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.715069 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:08:11.715086 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:08:11.715102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:08:11.715159 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:08:11.718390 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:11.719082 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:11.719089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:11.719091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:11.719224 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:11.719555 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:11.719558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:11.719560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:11.719984 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:08:11.720207 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.720524 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.720750 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:08:11.721832 12084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:08:11.813676 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 11952
I20260927 05:08:11.830228 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 12095
Sep 27 05:08:11 dist-test-slave-hgs0 krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:08:11.846258 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:08:11.831053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.0.97.3:34063)
0927 05:08:11.831218 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:08:11.831256 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0927 05:08:11.831276 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:08:11.831407 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:08:11.831411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:08:11.831475 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:08:11.831574 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:08:11.833008 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.833793 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.835141 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:08:11.835574 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:08:11.835672 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:08:11.840300 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:08:11.840319 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:08:11.840336 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:08:11.840378 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:08:11.843378 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:11.844182 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:11.844195 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:11.844251 (+ 56us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:11.844381 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:08:11.844775 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:08:11.844780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:08:11.844782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:08:11.845255 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:08:11.845498 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 05:08:11.845781 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 05:08:11.845955 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:08:11.846951 12218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:08:11.941793 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 12095
I20260927 05:08:11.958810 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 11701
I20260927 05:08:11.960091 11763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:08:12.076566 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 11701
I20260927 05:08:12.101218 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 05:08:12.437355 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 05:08:12.437563 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 05:08:12.784055 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 05:08:12.784 UTC [11287] LOG: received smart shutdown request
2026-09-27 05:08:12.788 UTC [11287] LOG: background worker "logical replication launcher" (PID 11296) exited with exit code 1
2026-09-27 05:08:12.788 UTC [11291] LOG: shutting down
2026-09-27 05:08:12.991 UTC [11287] LOG: database system is shut down
2026-09-27T05:08:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52609 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:08:13 dist-test-slave-hgs0 krb5kdc[12282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:08:13 dist-test-slave-hgs0 krb5kdc[12282](info): set up 2 sockets
Sep 27 05:08:13 dist-test-slave-hgs0 krb5kdc[12282](info): commencing operation
krb5kdc: starting...
W20260927 05:08:14.296693 388 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:08:14 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:08:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:08:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260927 05:08:14.670773 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskEQLg3e/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/postgres -l logfile start
2026-09-27 05:08:17.541 UTC [12312] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 05:08:17.541 UTC [12312] LOG: listening on IPv4 address "127.0.97.20", port 55065
2026-09-27 05:08:17.546 UTC [12312] LOG: listening on Unix socket "/tmp/.s.PGSQL.55065"
2026-09-27 05:08:17.570 UTC [12315] LOG: database system was shut down at 2026-09-27 05:08:15 UTC
2026-09-27 05:08:17.575 UTC [12312] LOG: database system is ready to accept connections
I20260927 05:08:18.576048 388 mini_postgres.cc:96] Postgres bound to 55065
2026-09-27 05:08:18.594 UTC [12323] FATAL: database "slave" does not exist
127.0.97.20:55065 - accepting connections
I20260927 05:08:18.594966 388 mini_ranger.cc:161] Starting Ranger...
I20260927 05:08:18.623546 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 05:08:19.202051 388 mini_ranger.cc:87] Created ranger Postgres database
I20260927 05:08:19.202199 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-admin
2026-09-27 05:08:19,523 [I] DB FLAVOR :POSTGRES
2026-09-27 05:08:19,523 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:08:19,524 [I] Checking connection..
2026-09-27 05:08:19,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 05:08:19,814 [I] Checking connection passed.
2026-09-27 05:08:19,814 [I] --------- Verifying version history table ---------
2026-09-27 05:08:19,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:08:20,105 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:08:20,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:08:20,369 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 05:08:20,369 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 05:08:20,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 05:08:20.649 UTC [12387] WARNING: there is no transaction in progress
2026-09-27 05:08:20,656 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 05:08:20,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 05:08:20,946 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 05:08:20,946 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:08:20,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 05:08:21,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 05:08:21,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hgs0.c.gcp-upstream.internal','N') ;"
2026-09-27 05:08:21,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:08:22,057 [I] Table x_portal_user does not exist in database ranger
2026-09-27 05:08:22,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:08:22,350 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 05:08:22,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:08:22,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:08:22,910 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 05:08:22,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 05:08:24.021 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.038 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.055 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.073 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.092 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.116 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.187 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.196 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.213 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.225 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.238 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.257 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.269 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.279 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.289 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:24.300 UTC [12495] WARNING: there is no transaction in progress
2026-09-27 05:08:25,091 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 05:08:25,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 05:08:25,358 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 05:08:25,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 05:08:25,639 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 05:08:25,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 05:08:25,911 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 05:08:25,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 05:08:26,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 05:08:26,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hgs0.c.gcp-upstream.internal';"
2026-09-27 05:08:26,712 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 05:08:26,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 05:08:26,984 [I] DB_PATCHES have already been applied
I20260927 05:08:26.990392 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskEQLg3e/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 05:08:26.990527 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260927 05:08:27.007264 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:34587
Sep 27, 2026 5:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 5:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 5:08:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 5:08:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 5:08:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 05:08:28.263567 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.669s user 0.000s sys 0.047s
Sep 27, 2026 5:08:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 5:08:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 5:08:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 5:08:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 5:08:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 05:08:50.077447 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 05:08:50 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485730, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 05:08:50.267544 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 05:08:50.289767 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 05:08:51.294931 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 05:08:51.295058 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-kms
I20260927 05:08:51.295112 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:55065
2026-09-27 05:08:51,671 [I] DB FLAVOR :POSTGRES
2026-09-27 05:08:51,672 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 05:08:51,672 [I] Checking connection
2026-09-27 05:08:51,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 05:08:51,965 [I] connection success
2026-09-27 05:08:51,965 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 05:08:51,965 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 05:08:51,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 05:08:52,260 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 05:08:52,260 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 05:08:52,261 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 05:08:52,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:55065/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 05:08:52,569 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 05:08:53.157454 388 mini_ranger_kms.cc:310] Created kms service
I20260927 05:08:53.417341 388 mini_ranger_kms.cc:326] Created kudu user
I20260927 05:08:53.471256 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 05:08:53 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485730, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:08:53.960072 388 mini_ranger_kms.cc:384] Added ranger policy
I20260927 05:08:53.960417 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskEQLg3e/build/asan/bin/postgresql.jar:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskEQLg3e/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 05:08:53.960474 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260927 05:08:53.979135 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12676
I20260927 05:08:53.979286 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:54465
Sep 27, 2026 5:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 5:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 5:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 5:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 5:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 5:08:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 05:08:56.436789 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.169s user 0.065s sys 0.326s
I20260927 05:08:56.437084 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 05:08:56.437291 388 mini_ranger_kms.cc:401] 127.0.97.20:54465/kms/v1/keys
Sep 27 05:08:56 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790485736, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 05:08:56 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485736, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 05:08:57 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485736, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 27 05:08:59 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:08:59.992517 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33851
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33851
--ranger_config_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54465/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:00.345127 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:00.345497 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:00.345665 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:00.353026 12726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:09:00.353200 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:00.353313 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:00.353430 12726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:09:00.353526 12726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:09:00.363924 12726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:54465/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33851
--ranger_config_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:00.365350 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:00.366788 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:00.379510 12731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:00.379467 12732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:00.379774 12726 server_base.cc:1034] running on GCE node
W20260927 05:09:00.380339 12734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:00.381470 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:00.383244 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:00.384471 12726 hybrid_clock.cc:648] HybridClock initialized: now 1790485740384418 us; error 46 us; skew 500 ppm
I20260927 05:09:00.384868 12726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:00 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:00.388621 12726 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:09:00.390450 12726 webserver.cc:460] Webserver started at http://127.0.97.62:32903/ using document root <none> and password file <none>
I20260927 05:09:00.391229 12726 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:00.391350 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:00.391750 12726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:09:00 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:09:00.444111 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "75ebc6945e834979ac8975140308280d"
format_stamp: "Formatted at 2026-09-27 05:09:00 on dist-test-slave-hgs0"
server_key: "ff58a438cf08ff5aa3f392c5480f8d08"
server_key_iv: "dc981f820466064463e5faeb26167411"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:00.445700 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "75ebc6945e834979ac8975140308280d"
format_stamp: "Formatted at 2026-09-27 05:09:00 on dist-test-slave-hgs0"
server_key: "ff58a438cf08ff5aa3f392c5480f8d08"
server_key_iv: "dc981f820466064463e5faeb26167411"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:00.453518 12726 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 05:09:00.473456 12741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:00.475427 12726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:09:00.475664 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "75ebc6945e834979ac8975140308280d"
format_stamp: "Formatted at 2026-09-27 05:09:00 on dist-test-slave-hgs0"
server_key: "ff58a438cf08ff5aa3f392c5480f8d08"
server_key_iv: "dc981f820466064463e5faeb26167411"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:00.475987 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:00.497359 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:00.502054 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:00.502406 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:00.526158 12726 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33851
I20260927 05:09:00.526176 12792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33851 every 8 connection(s)
I20260927 05:09:00.527782 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:09:00.532387 12793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:00.532543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 12726
I20260927 05:09:00.532824 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:09:00.533684 388 external_mini_cluster.cc:1428] Setting key d5728e12e522d57089d9b8ef6225a722
I20260927 05:09:00.548444 12793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: Bootstrap starting.
I20260927 05:09:00.552301 12793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:00.553540 12793 log.cc:824] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:09:00 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:00.557734 12793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: No bootstrap required, opened a new log
I20260927 05:09:00.564328 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:00.536244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.97.62:33851)
0927 05:09:00.540883 (+ 4639us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:00.540907 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:00.541344 (+ 437us) server_negotiation.cc:407] Connection header received
0927 05:09:00.543387 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:00.543415 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:00.543888 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:00.544430 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:00.546023 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:00.547520 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:00.549188 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:00.550000 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:00.550261 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:00.555283 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:00.555294 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:00.555315 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:00.555362 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:00.558846 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:00.559883 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:00.559891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:00.559899 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:00.560182 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:00.560666 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:00.560671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:00.560673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:00.561305 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:00.561571 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:00.562426 (+ 855us) server_negotiation.cc:299] Negotiation successful
0927 05:09:00.563032 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4304,"thread_start_us":4020,"threads_started":1}
I20260927 05:09:00.568725 12793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260927 05:09:00.569180 12793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:00.569316 12793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ebc6945e834979ac8975140308280d, State: Initialized, Role: FOLLOWER
I20260927 05:09:00.569974 12793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260927 05:09:00.570233 12793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:09:00.570389 12793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:09:00.570585 12793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:00.573235 12793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260927 05:09:00.573805 12793 leader_election.cc:302] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75ebc6945e834979ac8975140308280d; no voters:
I20260927 05:09:00.574453 12793 leader_election.cc:288] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:09:00.574672 12798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:09:00.575564 12798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 1 LEADER]: Becoming Leader. State: Replica: 75ebc6945e834979ac8975140308280d, State: Running, Role: LEADER
I20260927 05:09:00.576311 12798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260927 05:09:00.576913 12793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:09:00.579823 12799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75ebc6945e834979ac8975140308280d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } } }
I20260927 05:09:00.579823 12800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ebc6945e834979ac8975140308280d. Latest consensus state: current_term: 1 leader_uuid: "75ebc6945e834979ac8975140308280d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } } }
I20260927 05:09:00.580411 12800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:00.580411 12799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:00.581938 12804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:09:00.589720 12804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:09:00.593924 12793 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 05:09:00.609879 12804 catalog_manager.cc:1347] Generated new cluster ID: 548621a5f0934d8582adad39a0960500
I20260927 05:09:00.610059 12804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:09:00.634338 12804 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:09:00.637020 12804 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:09:00.648824 12804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: Generated new TSK 0
I20260927 05:09:00.650247 12804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:09:01.444489 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:09:01 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790485741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 05:09:01.884915 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:01.885399 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:01.885587 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:01.897825 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:01.897962 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:01.898226 12838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:09:01.910750 12838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:54465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:01.912429 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:01.914157 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:01.932087 12849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:01.937749 12848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:01.941841 12851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:01.942261 12838 server_base.cc:1034] running on GCE node
I20260927 05:09:01.943100 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:01.953408 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:01.954838 12838 hybrid_clock.cc:648] HybridClock initialized: now 1790485741954718 us; error 117 us; skew 500 ppm
I20260927 05:09:01.955299 12838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:01 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:01.960146 12838 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:09:01.962970 12838 webserver.cc:460] Webserver started at http://127.0.97.1:46735/ using document root <none> and password file <none>
I20260927 05:09:01.964098 12838 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:01.964265 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:01.964692 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:09:01 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:09:01.989905 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "520d6490cddd4fe18deec30f98c54e3f"
format_stamp: "Formatted at 2026-09-27 05:09:01 on dist-test-slave-hgs0"
server_key: "bce355f54ddcc356e783b2c5301a34a3"
server_key_iv: "feb33c6c9e6f5b1725d1cee10875ad3e"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:01.991084 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "520d6490cddd4fe18deec30f98c54e3f"
format_stamp: "Formatted at 2026-09-27 05:09:01 on dist-test-slave-hgs0"
server_key: "bce355f54ddcc356e783b2c5301a34a3"
server_key_iv: "feb33c6c9e6f5b1725d1cee10875ad3e"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:02.014317 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.008s sys 0.012s
I20260927 05:09:02.039637 12858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:02.041887 12838 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 05:09:02.042117 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "520d6490cddd4fe18deec30f98c54e3f"
format_stamp: "Formatted at 2026-09-27 05:09:01 on dist-test-slave-hgs0"
server_key: "bce355f54ddcc356e783b2c5301a34a3"
server_key_iv: "feb33c6c9e6f5b1725d1cee10875ad3e"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:02.042413 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:02.072535 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:02.078680 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:02.079144 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:02.086766 12838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:02.095517 12838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:02.095729 12838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 05:09:02.096004 12838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:02.096138 12838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:09:02 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:09:02.549650 12838 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42741
I20260927 05:09:02.549799 12970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42741 every 8 connection(s)
I20260927 05:09:02.551677 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:09:02.560652 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 12838
I20260927 05:09:02.560904 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:09:02.561757 388 external_mini_cluster.cc:1428] Setting key 96c97fdf67f6e97ccda998ef1a301e89
Sep 27 05:09:02 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:02.594542 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:02.557115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58723 (local address 127.0.97.62:33851)
0927 05:09:02.573057 (+ 15942us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:02.573065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:02.573099 (+ 34us) server_negotiation.cc:407] Connection header received
0927 05:09:02.574024 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:02.574030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:02.574129 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:02.574217 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:02.576918 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:02.578134 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:02.580850 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:02.581506 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:02.581620 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:02.586584 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:02.586592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:02.586596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:02.586644 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:02.590474 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:02.591571 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:02.591578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:02.591581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:02.591757 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:02.592355 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:02.592360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:02.592364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:02.592719 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:02.592853 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:02.593912 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0927 05:09:02.594152 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15778,"thread_start_us":15614,"threads_started":1}
I20260927 05:09:02.596896 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:02.556233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.1:58723)
0927 05:09:02.568893 (+ 12660us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:02.568991 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:02.573453 (+ 4462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:02.574890 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:02.574905 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:02.575619 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:02.576622 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:02.576654 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:02.578390 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:02.578395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:02.580609 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:02.580645 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:02.582023 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:02.582030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:02.582343 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:02.583266 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:02.583299 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:02.586251 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:02.590718 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:02.590729 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:02.590753 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:02.591409 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:02.591922 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:02.591927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:02.591953 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:02.592212 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:02.592972 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:02.592980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:02.593442 (+ 462us) client_negotiation.cc:769] Sending connection context
0927 05:09:02.594791 (+ 1349us) client_negotiation.cc:240] Negotiation successful
0927 05:09:02.595383 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12104,"thread_start_us":23914,"threads_started":1}
I20260927 05:09:02.610677 12971 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260927 05:09:02.611114 12971 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:02.612142 12971 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260927 05:09:02.615240 12758 ts_manager.cc:194] Registered new tserver with Master: 520d6490cddd4fe18deec30f98c54e3f (127.0.97.1:42741)
I20260927 05:09:02.617553 12758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58723
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:09:02.667402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:09:02 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 05:09:03.198912 12977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:03.199261 12977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:03.199438 12977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:03.207847 12977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:03.208077 12977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:03.208287 12977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:09:03.219089 12977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:54465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:03.220819 12977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:03.222376 12977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:03.239236 12994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:03.240023 12992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:03.240339 12991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:03.241143 12977 server_base.cc:1034] running on GCE node
I20260927 05:09:03.242007 12977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:03.243343 12977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:03.244606 12977 hybrid_clock.cc:648] HybridClock initialized: now 1790485743244549 us; error 47 us; skew 500 ppm
I20260927 05:09:03.245035 12977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:03 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:03.249336 12977 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:09:03.251125 12977 webserver.cc:460] Webserver started at http://127.0.97.2:45795/ using document root <none> and password file <none>
I20260927 05:09:03.252034 12977 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:03.252194 12977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:03.252593 12977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:09:03 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:09:03.273059 12977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "be0e07cc5a424a55b961a817a4f008b0"
format_stamp: "Formatted at 2026-09-27 05:09:03 on dist-test-slave-hgs0"
server_key: "907d4058af0d76602e93cd89db2640f0"
server_key_iv: "0f42131bbb63b7946eccb0a917c43689"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:03.274353 12977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "be0e07cc5a424a55b961a817a4f008b0"
format_stamp: "Formatted at 2026-09-27 05:09:03 on dist-test-slave-hgs0"
server_key: "907d4058af0d76602e93cd89db2640f0"
server_key_iv: "0f42131bbb63b7946eccb0a917c43689"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:03.282558 12977 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260927 05:09:03.304245 13001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:03.306455 12977 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.007s sys 0.001s
I20260927 05:09:03.306694 12977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "be0e07cc5a424a55b961a817a4f008b0"
format_stamp: "Formatted at 2026-09-27 05:09:03 on dist-test-slave-hgs0"
server_key: "907d4058af0d76602e93cd89db2640f0"
server_key_iv: "0f42131bbb63b7946eccb0a917c43689"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:03.306965 12977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:03.361214 12977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:03.365814 12977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:03.366142 12977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:03.367244 12977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:03.368945 12977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:03.369057 12977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:03.369211 12977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:03.369282 12977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:03.407609 12977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45903
I20260927 05:09:03.407630 13113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45903 every 8 connection(s)
I20260927 05:09:03.409214 12977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:09:03.415704 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 12977
I20260927 05:09:03.415900 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:09:03.416397 388 external_mini_cluster.cc:1428] Setting key ba576a7285275c4a04b9e7a3f10c6ada
Sep 27 05:09:03 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:03.437628 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:03.413128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50005 (local address 127.0.97.62:33851)
0927 05:09:03.413693 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:03.413701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:03.414877 (+ 1176us) server_negotiation.cc:407] Connection header received
0927 05:09:03.417433 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:03.417440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:03.417535 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:03.417655 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:03.419568 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:03.420874 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:03.423410 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:03.424191 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:03.424346 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:03.428843 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:03.428854 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:03.428862 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:03.428914 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:03.433057 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:03.434145 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:03.434154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:03.434161 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:03.434348 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:03.434964 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:03.434974 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:03.434980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:03.435449 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:03.435618 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:03.436756 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0927 05:09:03.437103 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":267,"threads_started":1}
I20260927 05:09:03.439496 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:03.413772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.2:50005)
0927 05:09:03.414670 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:03.414725 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:03.417117 (+ 2392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:03.417965 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:03.417977 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:03.418556 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:03.419357 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:03.419382 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:03.421062 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:03.421068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:03.423231 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:03.423249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:03.424468 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:03.424475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:03.424767 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:03.425626 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:03.425656 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:03.428548 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:03.433245 (+ 4697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:03.433255 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:03.433275 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:03.433977 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:03.434509 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:03.434515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:03.434519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:03.434815 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:03.435628 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:03.435638 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:03.436171 (+ 533us) client_negotiation.cc:769] Sending connection context
0927 05:09:03.436562 (+ 391us) client_negotiation.cc:240] Negotiation successful
0927 05:09:03.437357 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":333,"threads_started":1}
I20260927 05:09:03.441679 13114 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:09:03.442114 13114 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:03.443121 13114 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:09:03.445147 12758 ts_manager.cc:194] Registered new tserver with Master: be0e07cc5a424a55b961a817a4f008b0 (127.0.97.2:45903)
I20260927 05:09:03.446354 12758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:09:03.476523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:09:03.622087 12971 heartbeater.cc:503] Master 127.0.97.62:33851 was elected leader, sending a full tablet report...
W20260927 05:09:03.831127 13120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:03.831504 13120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:03.831709 13120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:03.840147 13120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:03.840329 13120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:03.840546 13120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:09:03.851998 13120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:54465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:03.853430 13120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:03.854872 13120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:03.869642 13128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:03.869642 13125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:03.869650 13126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:03.871218 13120 server_base.cc:1034] running on GCE node
I20260927 05:09:03.871922 13120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:03.873088 13120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:03.874312 13120 hybrid_clock.cc:648] HybridClock initialized: now 1790485743874259 us; error 47 us; skew 500 ppm
I20260927 05:09:03.874627 13120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:03 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:03.877977 13120 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:09:03.879742 13120 webserver.cc:460] Webserver started at http://127.0.97.3:38515/ using document root <none> and password file <none>
I20260927 05:09:03.880584 13120 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:03.880692 13120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:03.881022 13120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 05:09:03 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:09:03.897399 13120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "f0b54ae60cab415b85c515af29923b7e"
format_stamp: "Formatted at 2026-09-27 05:09:03 on dist-test-slave-hgs0"
server_key: "1ae6cfe4292a7ad6f681fe342143f414"
server_key_iv: "6bb68248c3a1e24e79a046247a8b100b"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:03.898290 13120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f0b54ae60cab415b85c515af29923b7e"
format_stamp: "Formatted at 2026-09-27 05:09:03 on dist-test-slave-hgs0"
server_key: "1ae6cfe4292a7ad6f681fe342143f414"
server_key_iv: "6bb68248c3a1e24e79a046247a8b100b"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:03.904367 13120 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 05:09:03.923769 13135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:03.925688 13120 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 05:09:03.925943 13120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "f0b54ae60cab415b85c515af29923b7e"
format_stamp: "Formatted at 2026-09-27 05:09:03 on dist-test-slave-hgs0"
server_key: "1ae6cfe4292a7ad6f681fe342143f414"
server_key_iv: "6bb68248c3a1e24e79a046247a8b100b"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:03.926209 13120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:03.959772 13120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:03.964706 13120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:03.965092 13120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:03.966316 13120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:03.968194 13120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:03.968282 13120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:03.968456 13120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:03.968510 13120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:04.006279 13120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36899
I20260927 05:09:04.006409 13247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36899 every 8 connection(s)
I20260927 05:09:04.007777 13120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:09:04.012202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 13120
I20260927 05:09:04.012411 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:09:04.012953 388 external_mini_cluster.cc:1428] Setting key 30cce5ce030050fcdcabd41e0b69de3e
Sep 27 05:09:04 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:04.032063 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:04.010921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50781 (local address 127.0.97.62:33851)
0927 05:09:04.011513 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:04.011519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:04.013543 (+ 2024us) server_negotiation.cc:407] Connection header received
0927 05:09:04.015340 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:04.015346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:04.015488 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:04.015622 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:04.017541 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:04.018337 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:04.020145 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:04.020614 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:04.020767 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:04.024919 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:04.024925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:04.024929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:04.024972 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:04.028267 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:04.029150 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:04.029155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:04.029158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:04.029308 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:04.029935 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:04.029939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:04.029941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:04.030306 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:04.030480 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:04.031339 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 05:09:04.031609 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":260,"threads_started":1}
I20260927 05:09:04.033170 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:04.011624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.3:50781)
0927 05:09:04.012845 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:04.012915 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:04.015010 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:04.015923 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:04.015985 (+ 62us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:04.016637 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:04.017327 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:04.017348 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:04.018511 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:04.018516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:04.019893 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:04.019916 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:04.020878 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:04.020888 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:04.021172 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:04.022048 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:04.022084 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:04.024560 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:04.028442 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:04.028454 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:04.028478 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:04.028990 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:04.029444 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:04.029452 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:04.029457 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:04.029790 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:04.030458 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:04.030467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:04.030925 (+ 458us) client_negotiation.cc:769] Sending connection context
0927 05:09:04.031255 (+ 330us) client_negotiation.cc:240] Negotiation successful
0927 05:09:04.031903 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":672,"threads_started":1}
I20260927 05:09:04.034622 13248 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260927 05:09:04.034974 13248 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:04.035737 13248 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260927 05:09:04.037330 12758 ts_manager.cc:194] Registered new tserver with Master: f0b54ae60cab415b85c515af29923b7e (127.0.97.3:36899)
I20260927 05:09:04.038179 12758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50781
I20260927 05:09:04.041834 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:09:04.043594 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 12838
Sep 27 05:09:04 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:09:04.062475 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:04.044700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.0.97.1:42741)
0927 05:09:04.045400 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:04.045408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:04.045437 (+ 29us) server_negotiation.cc:407] Connection header received
0927 05:09:04.045504 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:04.045510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:04.045737 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:04.045871 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:04.047635 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:04.049024 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:04.050780 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:04.051446 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:04.051562 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:04.056490 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:04.056506 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:04.056524 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:04.056571 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:04.059444 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:04.060419 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:04.060427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:04.060430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:04.060628 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:04.060964 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:04.060967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:04.060969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:04.061336 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:04.061576 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:04.061860 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 05:09:04.062074 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":374,"threads_started":1}
I20260927 05:09:04.064037 12966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:04.155112 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 12838
I20260927 05:09:04.172948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 12977
Sep 27 05:09:04 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:09:04.193082 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:04.174233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.0.97.2:45903)
0927 05:09:04.174897 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:04.174905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:04.174936 (+ 31us) server_negotiation.cc:407] Connection header received
0927 05:09:04.175005 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:04.175012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:04.175261 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:04.175420 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:04.177314 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:04.178538 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:04.180247 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:04.180913 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:04.181059 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:04.186304 (+ 5245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:04.186333 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:04.186351 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:04.186407 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:04.189958 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:04.190860 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:04.190869 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:04.190872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:04.191034 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:04.191412 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:04.191415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:04.191417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:04.191785 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:04.192099 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:04.192492 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 05:09:04.192718 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":342,"threads_started":1}
I20260927 05:09:04.194449 13109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:04.284792 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 12977
I20260927 05:09:04.307367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 13120
Sep 27 05:09:04 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:09:04.328842 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:04.308625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.0.97.3:36899)
0927 05:09:04.309408 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:04.309415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:04.309446 (+ 31us) server_negotiation.cc:407] Connection header received
0927 05:09:04.309511 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:04.309518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:04.309753 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:04.309927 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:04.311419 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:04.312732 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:04.314350 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:04.315083 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:04.315204 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:04.320961 (+ 5757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:04.320981 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:04.321001 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:04.321056 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:04.325495 (+ 4439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:04.326453 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:04.326462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:04.326465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:04.326632 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:04.327080 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:04.327084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:04.327086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:04.327492 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:04.327786 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:04.328216 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 05:09:04.328472 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":421,"threads_started":1}
I20260927 05:09:04.330428 13243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:04.421933 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 13120
I20260927 05:09:04.438357 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 12726
I20260927 05:09:04.439646 12788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:04.556660 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 12726
I20260927 05:09:04.577941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33851
--webserver_interface=127.0.97.62
--webserver_port=32903
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33851
--ranger_config_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54465/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:54465/kms with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:04.930569 13260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:04.930949 13260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:04.931123 13260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:04.939118 13260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:09:04.939308 13260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:04.939419 13260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:04.939539 13260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:09:04.939623 13260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:09:04.951267 13260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:54465/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33851
--ranger_config_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=32903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:04.952744 13260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:04.954258 13260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:04.967811 13266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:04.967897 13265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:04.967876 13268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:04.969182 13260 server_base.cc:1034] running on GCE node
I20260927 05:09:04.970206 13260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:04.972054 13260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:04.973311 13260 hybrid_clock.cc:648] HybridClock initialized: now 1790485744973247 us; error 61 us; skew 500 ppm
I20260927 05:09:04.973721 13260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:04 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:04.982677 13260 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:09:04.984608 13260 webserver.cc:460] Webserver started at http://127.0.97.62:32903/ using document root <none> and password file <none>
I20260927 05:09:04.985368 13260 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:04.985536 13260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:09:05 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:09:05.055511 13260 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:09:05.059805 13276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:05.061702 13260 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:09:05.062062 13260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "75ebc6945e834979ac8975140308280d"
format_stamp: "Formatted at 2026-09-27 05:09:00 on dist-test-slave-hgs0"
server_key: "ff58a438cf08ff5aa3f392c5480f8d08"
server_key_iv: "dc981f820466064463e5faeb26167411"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:05.063350 13260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:05.081648 13260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:05.087473 13260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:05.087872 13260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:05.107910 13260 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33851
I20260927 05:09:05.107986 13327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33851 every 8 connection(s)
I20260927 05:09:05.109627 13260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:09:05.114069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 13260
I20260927 05:09:05.114928 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42741
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33851
--webserver_port=46735
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:54465/kms with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:09:05.117192 13328 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:09:05.120927 13328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: Bootstrap starting.
I20260927 05:09:05.141925 13328 log.cc:824] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:05.152248 13328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:09:05.152944 13328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: Bootstrap complete.
I20260927 05:09:05.162855 13328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260927 05:09:05.163800 13328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ebc6945e834979ac8975140308280d, State: Initialized, Role: FOLLOWER
I20260927 05:09:05.164608 13328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260927 05:09:05.164882 13328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:09:05.165010 13328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:09:05.165210 13328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:09:05.169777 13328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260927 05:09:05.170328 13328 leader_election.cc:302] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75ebc6945e834979ac8975140308280d; no voters:
I20260927 05:09:05.171048 13328 leader_election.cc:288] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:09:05.171265 13332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:09:05.172333 13332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [term 2 LEADER]: Becoming Leader. State: Replica: 75ebc6945e834979ac8975140308280d, State: Running, Role: LEADER
I20260927 05:09:05.172922 13332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } }
I20260927 05:09:05.173448 13328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:09:05.175420 13334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ebc6945e834979ac8975140308280d. Latest consensus state: current_term: 2 leader_uuid: "75ebc6945e834979ac8975140308280d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } } }
I20260927 05:09:05.175911 13334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:05.176527 13333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "75ebc6945e834979ac8975140308280d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ebc6945e834979ac8975140308280d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33851 } } }
I20260927 05:09:05.176858 13333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:05.180526 13338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:09:05.194120 13338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:09:05.196076 13338 catalog_manager.cc:1259] Loaded cluster ID: 548621a5f0934d8582adad39a0960500
I20260927 05:09:05.196259 13338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:09:05.201655 13338 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:09:05.203688 13338 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: Loaded TSK: 0
I20260927 05:09:05.205740 13338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:09:05.218065 13328 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 05:09:05.603006 13330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:05.603458 13330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:05.603645 13330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:05.616560 13330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:05.616734 13330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:05.616958 13330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:09:05.634411 13330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:54465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46735
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:05.636520 13330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:05.638609 13330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:05.701620 13366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:05.704454 13367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:05.707170 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:05.707705 13330 server_base.cc:1034] running on GCE node
I20260927 05:09:05.708804 13330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:05.713457 13330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:05.714900 13330 hybrid_clock.cc:648] HybridClock initialized: now 1790485745714805 us; error 81 us; skew 500 ppm
I20260927 05:09:05.715430 13330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:05 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:05.720774 13330 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:09:05.742380 13330 webserver.cc:460] Webserver started at http://127.0.97.1:46735/ using document root <none> and password file <none>
I20260927 05:09:05.743588 13330 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:05.743791 13330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:09:05 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:09:05.782455 13330 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.001s
I20260927 05:09:05.788324 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:05.790535 13330 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260927 05:09:05.790854 13330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "520d6490cddd4fe18deec30f98c54e3f"
format_stamp: "Formatted at 2026-09-27 05:09:01 on dist-test-slave-hgs0"
server_key: "bce355f54ddcc356e783b2c5301a34a3"
server_key_iv: "feb33c6c9e6f5b1725d1cee10875ad3e"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:05.792505 13330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:05.833292 13330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:05.839120 13330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:05.839483 13330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:05.845937 13330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:05.847990 13330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:05.848136 13330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 05:09:05.848264 13330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:05.848343 13330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:05.925127 13330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42741
I20260927 05:09:05.925177 13490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42741 every 8 connection(s)
I20260927 05:09:05.926954 13330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:09:05.934821 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 13330
I20260927 05:09:05.935611 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45903
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33851
--webserver_port=45795
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:54465/kms with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:09:05 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:05.966849 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:05.931835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33953 (local address 127.0.97.62:33851)
0927 05:09:05.934229 (+ 2394us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:05.934248 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:05.934301 (+ 53us) server_negotiation.cc:407] Connection header received
0927 05:09:05.937024 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:05.937046 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:05.937599 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:05.938102 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:05.942632 (+ 4530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:05.944865 (+ 2233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:05.947896 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:05.949176 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:05.949425 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:05.954495 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:05.954507 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:05.954533 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:05.954593 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:05.959247 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:05.960696 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:05.960708 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:05.960719 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:05.960984 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:05.961710 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:05.961716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:05.961720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:05.962538 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:05.962769 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:05.964747 (+ 1978us) server_negotiation.cc:299] Negotiation successful
0927 05:09:05.965248 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2079,"thread_start_us":4656,"threads_started":1}
I20260927 05:09:05.968344 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:05.931008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.1:33953)
0927 05:09:05.932082 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:05.932150 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:05.936308 (+ 4158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:05.940560 (+ 4252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:05.940575 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:05.941336 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:05.942273 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:05.942304 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:05.945106 (+ 2802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:05.945116 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:05.947659 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:05.947683 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:05.949489 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:05.949500 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:05.949857 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:05.950851 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:05.950906 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:05.954106 (+ 3200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:05.959503 (+ 5397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:05.959519 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:05.959547 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:05.960443 (+ 896us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:05.961169 (+ 726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:05.961178 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:05.961184 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:05.961524 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:05.962740 (+ 1216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:05.962754 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:05.964064 (+ 1310us) client_negotiation.cc:769] Sending connection context
0927 05:09:05.964421 (+ 357us) client_negotiation.cc:240] Negotiation successful
0927 05:09:05.965093 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":1081,"threads_started":1}
I20260927 05:09:05.972883 13491 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260927 05:09:05.973274 13491 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:05.974313 13491 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
W20260927 05:09:05.977986 13491 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33851 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 05:09:05.978229 13491 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:05.978534 13491 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260927 05:09:05.980016 13491 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:05.980398 13491 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
W20260927 05:09:05.981619 13491 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 05:09:06.415175 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:06.415488 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:06.415644 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:06.427193 13495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:06.427379 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:06.427592 13495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:09:06.441730 13495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:54465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45795
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:06.443385 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:06.445127 13495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:06.488260 13509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:06.489377 13510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:06.492808 13512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:06.493822 13495 server_base.cc:1034] running on GCE node
I20260927 05:09:06.494768 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:06.496663 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:06.498019 13495 hybrid_clock.cc:648] HybridClock initialized: now 1790485746497890 us; error 81 us; skew 500 ppm
I20260927 05:09:06.498502 13495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:06 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:06.504742 13495 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:09:06.514806 13495 webserver.cc:460] Webserver started at http://127.0.97.2:45795/ using document root <none> and password file <none>
I20260927 05:09:06.515816 13495 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:06.516043 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:09:06 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:09:06.541102 13495 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 05:09:06.544416 13522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:06.546105 13495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 05:09:06.546329 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "be0e07cc5a424a55b961a817a4f008b0"
format_stamp: "Formatted at 2026-09-27 05:09:03 on dist-test-slave-hgs0"
server_key: "907d4058af0d76602e93cd89db2640f0"
server_key_iv: "0f42131bbb63b7946eccb0a917c43689"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:06.547458 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:06.567673 13495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:06.573125 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:06.573474 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:06.574731 13495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:06.576603 13495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:06.576752 13495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:06.576907 13495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:06.577013 13495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 05:09:06 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790485746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:06.670009 13495 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45903
I20260927 05:09:06.670049 13635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45903 every 8 connection(s)
I20260927 05:09:06.671743 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:09:06.677805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 13495
I20260927 05:09:06.678880 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:54465/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:36899
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33851
--webserver_port=38515
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:54465/kms with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:09:06 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:06.747159 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:06.680261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41451 (local address 127.0.97.62:33851)
0927 05:09:06.711885 (+ 31624us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:06.711892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:06.714798 (+ 2906us) server_negotiation.cc:407] Connection header received
0927 05:09:06.724134 (+ 9336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:06.724141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:06.724255 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:06.724354 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:06.727383 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:06.728752 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:06.732046 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:06.732849 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:06.732978 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:06.738405 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:06.738414 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:06.738419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:06.738501 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:06.742461 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:06.743781 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:06.743789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:06.743793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:06.743995 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:06.744655 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:06.744661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:06.744664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:06.745060 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:06.745200 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:06.746448 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0927 05:09:06.746736 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31424,"thread_start_us":31654,"threads_started":1}
I20260927 05:09:06.750339 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:06.676525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.2:41451)
0927 05:09:06.713053 (+ 36528us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:06.713124 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:06.723012 (+ 9888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:06.725091 (+ 2079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:06.725105 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:06.725977 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:06.727069 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:06.727104 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:06.729028 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:06.729035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:06.731697 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:06.731730 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:06.733468 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:06.733476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:06.733849 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:06.734785 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:06.734833 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:06.738017 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:06.742735 (+ 4718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:06.742747 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:06.742773 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:06.743569 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:06.744163 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:06.744168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:06.744173 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:06.744491 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:06.745332 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:06.745342 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:06.745888 (+ 546us) client_negotiation.cc:769] Sending connection context
0927 05:09:06.747868 (+ 1980us) client_negotiation.cc:240] Negotiation successful
0927 05:09:06.748623 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36064,"thread_start_us":43296,"threads_started":1}
I20260927 05:09:06.753116 13636 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260927 05:09:06.753546 13636 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:06.754683 13636 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260927 05:09:06.757414 13293 ts_manager.cc:194] Registered new tserver with Master: be0e07cc5a424a55b961a817a4f008b0 (127.0.97.2:45903)
I20260927 05:09:06.760004 13293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41451
I20260927 05:09:06.981782 13491 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:06.982280 13491 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260927 05:09:06.983637 13293 ts_manager.cc:194] Registered new tserver with Master: 520d6490cddd4fe18deec30f98c54e3f (127.0.97.1:42741)
I20260927 05:09:06.984907 13293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33953
Sep 27 05:09:07 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260927 05:09:07.271613 13640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:07.272114 13640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:07.272301 13640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:07.284871 13640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:07.285043 13640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:07.285226 13640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:09:07.302450 13640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:54465/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:36899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38515
--tserver_master_addrs=127.0.97.62:33851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:07.304334 13640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:07.306180 13640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:07.356673 13645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:07.356840 13646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:07.356895 13649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:07.357501 13640 server_base.cc:1034] running on GCE node
I20260927 05:09:07.358803 13640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:07.366796 13640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:07.370316 13640 hybrid_clock.cc:648] HybridClock initialized: now 1790485747370235 us; error 117 us; skew 500 ppm
I20260927 05:09:07.370808 13640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:07 dist-test-slave-hgs0 krb5kdc[12282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:07.375808 13640 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:09:07.396539 13640 webserver.cc:460] Webserver started at http://127.0.97.3:38515/ using document root <none> and password file <none>
I20260927 05:09:07.397715 13640 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:07.398029 13640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 05:09:07 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:09:07.457525 13640 fs_manager.cc:709] Time spent opening directory manager: real 0.028s user 0.010s sys 0.017s
I20260927 05:09:07.468331 13657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:07.470551 13640 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.003s
I20260927 05:09:07.470845 13640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "f0b54ae60cab415b85c515af29923b7e"
format_stamp: "Formatted at 2026-09-27 05:09:03 on dist-test-slave-hgs0"
server_key: "1ae6cfe4292a7ad6f681fe342143f414"
server_key_iv: "6bb68248c3a1e24e79a046247a8b100b"
server_key_version: "kuduclusterkey@0"
I20260927 05:09:07.472741 13640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 05:09:07 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260927 05:09:07.555543 13640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:07.561395 13640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:07.561733 13640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:07.569481 13640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:07.578044 13640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:07.578199 13640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:07.578337 13640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:07.578418 13640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:07.766039 13636 heartbeater.cc:503] Master 127.0.97.62:33851 was elected leader, sending a full tablet report...
I20260927 05:09:07.770802 13640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36899
I20260927 05:09:07.770915 13769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36899 every 8 connection(s)
I20260927 05:09:07.772701 13640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:09:07.780176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 13640
Sep 27 05:09:07 dist-test-slave-hgs0 krb5kdc[12282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:07.836745 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:07.781348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.0.97.62:33851)
0927 05:09:07.816592 (+ 35244us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:07.816602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:07.817236 (+ 634us) server_negotiation.cc:407] Connection header received
0927 05:09:07.817470 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:07.817475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:07.817572 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:07.817683 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:07.821644 (+ 3961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.822800 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.824776 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.825344 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.825495 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:07.828840 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:07.828846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:07.828852 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:07.828893 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:07.832694 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:07.833631 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:07.833637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:07.833642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:07.833803 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:07.834269 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:07.834275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:07.834279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:07.834668 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:07.834938 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:07.836025 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0927 05:09:07.836261 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35015,"thread_start_us":7107,"threads_started":1}
I20260927 05:09:07.838861 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:07.775808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52895 (local address 127.0.97.62:33851)
0927 05:09:07.788749 (+ 12941us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:07.788756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:07.788787 (+ 31us) server_negotiation.cc:407] Connection header received
0927 05:09:07.818821 (+ 30034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:07.818826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:07.818909 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:07.819015 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:07.823144 (+ 4129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.824445 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.826481 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.827153 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.827279 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:07.831409 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:07.831416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:07.831420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:07.831475 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:07.834623 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:07.835442 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:07.835449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:07.835452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:07.835583 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:07.836227 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:07.836232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:07.836236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:07.836570 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:07.836702 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:07.838209 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0927 05:09:07.838457 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12779,"thread_start_us":40327,"threads_started":1}
I20260927 05:09:07.840845 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:07.776500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33851 (local address 127.0.97.3:52895)
0927 05:09:07.785715 (+ 9215us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:07.785816 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:07.818501 (+ 32685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:07.819608 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:07.819624 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:07.820407 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:07.821456 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:07.821485 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.824638 (+ 3153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.824645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:07.826325 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:07.826346 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.827437 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.827447 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:07.827795 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:07.828620 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:07.828652 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:07.831144 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:07.834763 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:07.834773 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:07.834792 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:07.835313 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:07.835727 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:07.835733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:07.835737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:07.836079 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:07.836814 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:07.837115 (+ 301us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:07.837821 (+ 706us) client_negotiation.cc:769] Sending connection context
0927 05:09:07.838108 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 05:09:07.839670 (+ 1562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8718,"thread_start_us":12584,"threads_started":1}
I20260927 05:09:07.842440 13770 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33851
I20260927 05:09:07.842844 13770 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:07.843676 13770 heartbeater.cc:511] Master 127.0.97.62:33851 requested a full tablet report, sending...
I20260927 05:09:07.845341 13293 ts_manager.cc:194] Registered new tserver with Master: f0b54ae60cab415b85c515af29923b7e (127.0.97.3:36899)
I20260927 05:09:07.846361 13293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52895
I20260927 05:09:07.852141 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:09:07.868005 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:07.855265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43760 (local address 127.0.97.62:33851)
0927 05:09:07.855519 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:07.855527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:07.855822 (+ 295us) server_negotiation.cc:407] Connection header received
0927 05:09:07.856009 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:07.856013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:07.856075 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:07.856155 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:07.857626 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.858429 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.859796 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.860278 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.860392 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:07.862887 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:07.862892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:07.862895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:07.862937 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:07.865397 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:07.866079 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:07.866083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:07.866085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:07.866204 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:07.866578 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:07.866581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:07.866583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:07.866839 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:07.867038 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:07.867391 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 05:09:07.867596 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:09:07.872956 13293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:09:07.880754 13293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:09:07.906554 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:07.895461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.0.97.2:45903)
0927 05:09:07.896673 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:07.896682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:07.896709 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:09:07.896822 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:07.896832 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:07.897171 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:07.898269 (+ 1098us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:07.898595 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.901104 (+ 2509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.903988 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.905161 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.905324 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:07.905576 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:07.906053 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 05:09:07.906232 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":1583,"threads_started":1}
I20260927 05:09:07.906629 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:07.894997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45903 (local address 127.0.0.1:44698)
0927 05:09:07.895863 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:07.895909 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:07.896102 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:07.897491 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:07.897500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:07.897530 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:07.898056 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:07.898070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.901270 (+ 3200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.901279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:07.903758 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:07.903774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.905393 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.905399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:07.905633 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:07.905760 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 05:09:07.906005 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 05:09:07.906143 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":73,"spinlock_wait_cycles":5376,"thread_start_us":627,"threads_started":1}
I20260927 05:09:07.907222 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:07.895640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.0.97.3:36899)
0927 05:09:07.898319 (+ 2679us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:07.898330 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:07.898366 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:09:07.898458 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:07.898465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:07.898713 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:07.899477 (+ 764us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:07.899768 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.901034 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.903747 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.905193 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.906304 (+ 1111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:07.906516 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:07.906663 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 05:09:07.906863 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2497,"thread_start_us":2333,"threads_started":1}
I20260927 05:09:07.907744 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:07.898463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.0.97.1:42741)
0927 05:09:07.899196 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:07.899202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:07.899224 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:09:07.899294 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:07.899299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:07.899458 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:07.899679 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:07.900437 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.901731 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.904603 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.906064 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.906282 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:07.906513 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:07.907238 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 05:09:07.907436 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":442,"threads_started":1}
I20260927 05:09:07.908224 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:07.897828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:43032)
0927 05:09:07.898562 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:07.898580 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:07.898689 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:07.899667 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:07.899671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:07.899696 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:07.900166 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:07.900178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.901855 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.901859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:07.904426 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:07.904440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.906826 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.906834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:07.907032 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:07.907068 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:09:07.907171 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 05:09:07.907896 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":772,"threads_started":1}
I20260927 05:09:07.907114 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:07.895049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36899 (local address 127.0.0.1:56220)
0927 05:09:07.897597 (+ 2548us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:07.897618 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:07.897756 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:07.898928 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:07.898932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:07.898960 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:07.899274 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:07.899287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.901183 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.901190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:07.903536 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:07.903549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:07.905454 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:07.905460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:07.905683 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:07.905768 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 05:09:07.905866 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 05:09:07.906064 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2420,"mutex_wait_us":397,"spinlock_wait_cycles":8320,"thread_start_us":2286,"threads_started":1}
I20260927 05:09:07.910878 13570 tablet_service.cc:1467] Processing CreateTablet for tablet a8d34480d7504ffbba9ebf164ae812a7 (DEFAULT_TABLE table=test-table [id=0ad93a5deb3644229ec82ae938e688b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:07.911314 13705 tablet_service.cc:1467] Processing CreateTablet for tablet a8d34480d7504ffbba9ebf164ae812a7 (DEFAULT_TABLE table=test-table [id=0ad93a5deb3644229ec82ae938e688b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:07.912307 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d34480d7504ffbba9ebf164ae812a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:07.912590 13705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d34480d7504ffbba9ebf164ae812a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:07.914925 13426 tablet_service.cc:1467] Processing CreateTablet for tablet a8d34480d7504ffbba9ebf164ae812a7 (DEFAULT_TABLE table=test-table [id=0ad93a5deb3644229ec82ae938e688b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:07.916291 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8d34480d7504ffbba9ebf164ae812a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:07.934998 13798 tablet_bootstrap.cc:492] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: Bootstrap starting.
I20260927 05:09:07.936107 13799 tablet_bootstrap.cc:492] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: Bootstrap starting.
I20260927 05:09:07.937218 13800 tablet_bootstrap.cc:492] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f: Bootstrap starting.
I20260927 05:09:07.939500 13798 tablet_bootstrap.cc:654] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:07.939976 13800 tablet_bootstrap.cc:654] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:07.940745 13798 log.cc:824] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:07.941174 13800 log.cc:824] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:07.944324 13798 tablet_bootstrap.cc:492] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: No bootstrap required, opened a new log
I20260927 05:09:07.944321 13800 tablet_bootstrap.cc:492] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f: No bootstrap required, opened a new log
I20260927 05:09:07.944444 13799 tablet_bootstrap.cc:654] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:07.944624 13798 ts_tablet_manager.cc:1397] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 05:09:07.944623 13800 ts_tablet_manager.cc:1397] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 05:09:07.945502 13799 log.cc:824] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:07.948273 13799 tablet_bootstrap.cc:492] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: No bootstrap required, opened a new log
I20260927 05:09:07.948558 13799 ts_tablet_manager.cc:1397] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260927 05:09:07.956740 13800 raft_consensus.cc:348] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 } } peers { permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "520d6490cddd4fe18deec30f98c54e3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } }
I20260927 05:09:07.956739 13798 raft_consensus.cc:348] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 } } peers { permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "520d6490cddd4fe18deec30f98c54e3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } }
I20260927 05:09:07.957316 13798 raft_consensus.cc:374] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:07.957317 13800 raft_consensus.cc:374] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:07.957438 13798 raft_consensus.cc:729] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0e07cc5a424a55b961a817a4f008b0, State: Initialized, Role: FOLLOWER
I20260927 05:09:07.957443 13800 raft_consensus.cc:729] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 520d6490cddd4fe18deec30f98c54e3f, State: Initialized, Role: FOLLOWER
I20260927 05:09:07.957422 13799 raft_consensus.cc:348] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 } } peers { permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "520d6490cddd4fe18deec30f98c54e3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } }
I20260927 05:09:07.957926 13799 raft_consensus.cc:374] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:07.958043 13799 raft_consensus.cc:729] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0b54ae60cab415b85c515af29923b7e, State: Initialized, Role: FOLLOWER
I20260927 05:09:07.958222 13798 consensus_queue.cc:260] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 } } peers { permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "520d6490cddd4fe18deec30f98c54e3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } }
I20260927 05:09:07.958222 13800 consensus_queue.cc:260] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 } } peers { permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "520d6490cddd4fe18deec30f98c54e3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } }
I20260927 05:09:07.958730 13799 consensus_queue.cc:260] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 } } peers { permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "520d6490cddd4fe18deec30f98c54e3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } }
I20260927 05:09:07.960007 13491 heartbeater.cc:503] Master 127.0.97.62:33851 was elected leader, sending a full tablet report...
I20260927 05:09:07.960425 13800 ts_tablet_manager.cc:1428] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260927 05:09:07.960417 13798 ts_tablet_manager.cc:1428] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260927 05:09:07.962464 13770 heartbeater.cc:503] Master 127.0.97.62:33851 was elected leader, sending a full tablet report...
I20260927 05:09:07.964869 13799 ts_tablet_manager.cc:1428] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260927 05:09:07.979794 13637 tablet.cc:2369] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:09:08.056839 13804 raft_consensus.cc:482] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:09:08.057161 13804 raft_consensus.cc:504] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 } } peers { permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "520d6490cddd4fe18deec30f98c54e3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } }
I20260927 05:09:08.058938 13804 leader_election.cc:288] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be0e07cc5a424a55b961a817a4f008b0 (127.0.97.2:45903), f0b54ae60cab415b85c515af29923b7e (127.0.97.3:36899)
I20260927 05:09:08.067639 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.059482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37625 (local address 127.0.97.2:45903)
0927 05:09:08.059705 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:08.059713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:08.060214 (+ 501us) server_negotiation.cc:407] Connection header received
0927 05:09:08.060446 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:08.060451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:08.060542 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:08.060648 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:08.061465 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.062615 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.065828 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.066656 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.066865 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:08.066983 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:08.067199 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 05:09:08.067364 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:09:08.067840 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.059206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45903 (local address 127.0.97.1:37625)
0927 05:09:08.060071 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:08.060092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:08.060264 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:08.060677 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:08.060682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:08.060710 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:08.061332 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:08.061341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.062760 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.062765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:08.065609 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:08.065629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.066903 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.066908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:08.067024 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:08.067051 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 05:09:08.067162 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 05:09:08.067325 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":280,"thread_start_us":497,"threads_started":1}
I20260927 05:09:08.068935 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.059464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37749 (local address 127.0.97.3:36899)
0927 05:09:08.059701 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:08.059711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:08.060313 (+ 602us) server_negotiation.cc:407] Connection header received
0927 05:09:08.060454 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:08.060458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:08.060542 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:08.060648 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:08.061464 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.062595 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.066940 (+ 4345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.068067 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.068232 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:08.068368 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:08.068612 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 05:09:08.068749 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 05:09:08.069103 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.059357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36899 (local address 127.0.97.1:37749)
0927 05:09:08.060210 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:08.060231 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:08.060348 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:08.060677 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:08.060682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:08.060776 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:08.061257 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:08.061279 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.062760 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.062765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:08.065609 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:08.065629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.068315 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.068320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:08.068457 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:08.068481 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 05:09:08.068560 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 05:09:08.068672 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":202,"spinlock_wait_cycles":25856,"thread_start_us":529,"threads_started":1}
I20260927 05:09:08.069262 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d34480d7504ffbba9ebf164ae812a7" candidate_uuid: "520d6490cddd4fe18deec30f98c54e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0e07cc5a424a55b961a817a4f008b0" is_pre_election: true
I20260927 05:09:08.069904 13590 raft_consensus.cc:2454] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 520d6490cddd4fe18deec30f98c54e3f in term 0.
I20260927 05:09:08.070158 13725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d34480d7504ffbba9ebf164ae812a7" candidate_uuid: "520d6490cddd4fe18deec30f98c54e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0b54ae60cab415b85c515af29923b7e" is_pre_election: true
I20260927 05:09:08.070665 13379 leader_election.cc:302] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 520d6490cddd4fe18deec30f98c54e3f, be0e07cc5a424a55b961a817a4f008b0; no voters:
I20260927 05:09:08.070722 13725 raft_consensus.cc:2454] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 520d6490cddd4fe18deec30f98c54e3f in term 0.
I20260927 05:09:08.071085 13804 raft_consensus.cc:2781] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:09:08.071260 13804 raft_consensus.cc:482] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:09:08.071393 13804 raft_consensus.cc:3037] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 0 FOLLOWER]: Advancing to term 1
W20260927 05:09:08.071902 13771 tablet.cc:2369] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:09:08.074230 13804 raft_consensus.cc:504] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 } } peers { permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "520d6490cddd4fe18deec30f98c54e3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } }
I20260927 05:09:08.074944 13804 leader_election.cc:288] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [CANDIDATE]: Term 1 election: Requested vote from peers be0e07cc5a424a55b961a817a4f008b0 (127.0.97.2:45903), f0b54ae60cab415b85c515af29923b7e (127.0.97.3:36899)
I20260927 05:09:08.075461 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d34480d7504ffbba9ebf164ae812a7" candidate_uuid: "520d6490cddd4fe18deec30f98c54e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be0e07cc5a424a55b961a817a4f008b0"
I20260927 05:09:08.075552 13725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8d34480d7504ffbba9ebf164ae812a7" candidate_uuid: "520d6490cddd4fe18deec30f98c54e3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0b54ae60cab415b85c515af29923b7e"
I20260927 05:09:08.075683 13590 raft_consensus.cc:3037] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:08.075819 13725 raft_consensus.cc:3037] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:08.078097 13590 raft_consensus.cc:2454] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 520d6490cddd4fe18deec30f98c54e3f in term 1.
I20260927 05:09:08.078167 13725 raft_consensus.cc:2454] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 520d6490cddd4fe18deec30f98c54e3f in term 1.
I20260927 05:09:08.078622 13379 leader_election.cc:302] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 520d6490cddd4fe18deec30f98c54e3f, be0e07cc5a424a55b961a817a4f008b0; no voters:
I20260927 05:09:08.078948 13804 raft_consensus.cc:2781] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:09:08.079443 13804 raft_consensus.cc:686] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 1 LEADER]: Becoming Leader. State: Replica: 520d6490cddd4fe18deec30f98c54e3f, State: Running, Role: LEADER
I20260927 05:09:08.080017 13804 consensus_queue.cc:237] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 } } peers { permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } } peers { permanent_uuid: "520d6490cddd4fe18deec30f98c54e3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } }
I20260927 05:09:08.085216 13293 catalog_manager.cc:5495] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 520d6490cddd4fe18deec30f98c54e3f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "520d6490cddd4fe18deec30f98c54e3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "520d6490cddd4fe18deec30f98c54e3f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } health_report { overall_health: HEALTHY } } }
I20260927 05:09:08.106649 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.100971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.0.97.1:42741)
0927 05:09:08.101190 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:08.101195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:08.101216 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:09:08.101388 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:08.101392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:08.101493 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:08.101639 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:09:08.102032 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.102846 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.104438 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.104975 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.105088 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:08.105568 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:09:08.106113 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:09:08.106311 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:08.106352 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 05:09:08.106426 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 05:09:08.115981 13590 raft_consensus.cc:1261] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0 [term 1 FOLLOWER]: Refusing update from remote peer 520d6490cddd4fe18deec30f98c54e3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:09:08.115995 13725 raft_consensus.cc:1261] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e [term 1 FOLLOWER]: Refusing update from remote peer 520d6490cddd4fe18deec30f98c54e3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:09:08.117187 13809 consensus_queue.cc:1035] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0b54ae60cab415b85c515af29923b7e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:09:08.117610 13804 consensus_queue.cc:1035] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0e07cc5a424a55b961a817a4f008b0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:09:08.136255 13813 mvcc.cc:205] Tried to move back new op lower bound from 7333829624271450112 to 7333829624140066816. Current Snapshot: MvccSnapshot[applied={T|T < 7333829624271450112}]
I20260927 05:09:08.141687 13814 mvcc.cc:205] Tried to move back new op lower bound from 7333829624271450112 to 7333829624140066816. Current Snapshot: MvccSnapshot[applied={T|T < 7333829624271450112}]
I20260927 05:09:08.153119 13293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:09:08.153442 13293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:09:08.157550 13293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75ebc6945e834979ac8975140308280d: Sending DeleteTablet for 3 replicas of tablet a8d34480d7504ffbba9ebf164ae812a7
I20260927 05:09:08.158912 13570 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d34480d7504ffbba9ebf164ae812a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44698
I20260927 05:09:08.159276 13705 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d34480d7504ffbba9ebf164ae812a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56220
I20260927 05:09:08.160022 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet a8d34480d7504ffbba9ebf164ae812a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43032
I20260927 05:09:08.165666 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 13330
I20260927 05:09:08.187419 13823 tablet_replica.cc:330] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f: stopping tablet replica
I20260927 05:09:08.188074 13823 raft_consensus.cc:2229] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:09:08.188944 13823 raft_consensus.cc:2258] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:08.191814 13823 ts_tablet_manager.cc:1905] T a8d34480d7504ffbba9ebf164ae812a7 P 520d6490cddd4fe18deec30f98c54e3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:09:08.196676 13821 tablet_replica.cc:330] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: stopping tablet replica
I20260927 05:09:08.197171 13821 raft_consensus.cc:2229] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:09:08.197609 13822 tablet_replica.cc:330] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: stopping tablet replica
I20260927 05:09:08.197741 13821 raft_consensus.cc:2258] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:08.198154 13822 raft_consensus.cc:2229] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:09:08.198467 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.172507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.0.97.1:42741)
0927 05:09:08.172753 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:08.172763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:08.172788 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:09:08.172868 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:08.172876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:08.172993 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:08.173118 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:08.175616 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.177371 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.180441 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.181570 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.181745 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:08.186919 (+ 5174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:08.186942 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:08.186987 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:08.187068 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:08.192377 (+ 5309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:08.193663 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:08.193676 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:08.193684 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:08.193896 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:08.194438 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:08.194447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:08.194451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:08.195066 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:08.195422 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:08.196896 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0927 05:09:08.197335 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:09:08.198698 13822 raft_consensus.cc:2258] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:08.199910 13486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:08.201761 13822 ts_tablet_manager.cc:1905] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:09:08.202523 13821 ts_tablet_manager.cc:1905] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:09:08.209694 13821 ts_tablet_manager.cc:1918] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:09:08.209872 13821 log.cc:1196] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/a8d34480d7504ffbba9ebf164ae812a7
I20260927 05:09:08.210363 13821 ts_tablet_manager.cc:1939] T a8d34480d7504ffbba9ebf164ae812a7 P be0e07cc5a424a55b961a817a4f008b0: Deleting consensus metadata
I20260927 05:09:08.211124 13822 ts_tablet_manager.cc:1918] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:09:08.211313 13822 log.cc:1196] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/a8d34480d7504ffbba9ebf164ae812a7
I20260927 05:09:08.211858 13822 ts_tablet_manager.cc:1939] T a8d34480d7504ffbba9ebf164ae812a7 P f0b54ae60cab415b85c515af29923b7e: Deleting consensus metadata
I20260927 05:09:08.212203 13277 catalog_manager.cc:4841] TS be0e07cc5a424a55b961a817a4f008b0 (127.0.97.2:45903): tablet a8d34480d7504ffbba9ebf164ae812a7 (table test-table [id=0ad93a5deb3644229ec82ae938e688b0]) successfully deleted
I20260927 05:09:08.214591 13280 catalog_manager.cc:4841] TS f0b54ae60cab415b85c515af29923b7e (127.0.97.3:36899): tablet a8d34480d7504ffbba9ebf164ae812a7 (table test-table [id=0ad93a5deb3644229ec82ae938e688b0]) successfully deleted
I20260927 05:09:08.309247 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 13330
I20260927 05:09:08.327762 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 13495
W20260927 05:09:08.327733 13280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42741 (error 108)
I20260927 05:09:08.329620 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.329120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:43058)
0927 05:09:08.329297 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:08.329435 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:09:08.329845 13280 catalog_manager.cc:4584] TS 520d6490cddd4fe18deec30f98c54e3f (127.0.97.1:42741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8d34480d7504ffbba9ebf164ae812a7: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
I20260927 05:09:08.341681 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.328739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.0.97.2:45903)
0927 05:09:08.328943 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:08.328949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:08.328970 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:09:08.329156 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:08.329164 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:08.329249 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:08.329435 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:08.330832 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.331697 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.333204 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.333601 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.333693 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:08.336048 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:08.336064 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:08.336081 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:08.336115 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:08.339241 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:08.339928 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:08.339973 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:08.339979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:08.340122 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:08.340431 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:08.340434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:08.340436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:08.340768 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:08.340994 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:08.341216 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 05:09:08.341380 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:09:08.342422 13631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:08.383613 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.383133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:43074)
0927 05:09:08.383378 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:08.383470 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 05:09:08.384704 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.384365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:43084)
0927 05:09:08.384518 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:08.384600 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 05:09:08.434654 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 13495
I20260927 05:09:08.445142 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.444667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:43094)
0927 05:09:08.444881 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:08.444973 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 05:09:08.446209 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.445860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:43098)
0927 05:09:08.446013 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:08.446093 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 05:09:08.452468 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 13640
I20260927 05:09:08.466032 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.453221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.0.97.3:36899)
0927 05:09:08.453392 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:08.453397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:08.453415 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:09:08.453510 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:08.453514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:08.453577 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:08.453657 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:08.454906 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.455636 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.457094 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:08.457510 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:08.457614 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:08.460141 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:08.460160 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:08.460176 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:08.460224 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:08.463169 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:08.463900 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:08.463908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:08.463911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:08.464130 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:08.464578 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:08.464582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:08.464584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:08.464989 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:08.465302 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:08.465591 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 05:09:08.465761 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:09:08.466709 13765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:08.526676 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.526195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:43102)
0927 05:09:08.526412 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:08.526500 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 05:09:08.527691 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:08.527316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:43108)
0927 05:09:08.527504 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:08.527580 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 05:09:08.556737 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 13640
I20260927 05:09:08.573252 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 13260
I20260927 05:09:08.574501 13323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:08.685993 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 13260
I20260927 05:09:08.710631 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 05:09:09.046430 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 05:09:09.046633 388 mini_ranger.cc:66] Stopping Ranger...
I20260927 05:09:09.392953 388 mini_ranger.cc:68] Stopped Ranger
2026-09-27 05:09:09.393 UTC [12312] LOG: received smart shutdown request
2026-09-27 05:09:09.396 UTC [12312] LOG: background worker "logical replication launcher" (PID 12321) exited with exit code 1
2026-09-27 05:09:09.397 UTC [12316] LOG: shutting down
2026-09-27 05:09:09.598 UTC [12312] LOG: database system is shut down
2026-09-27T05:09:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56636 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:09:09 dist-test-slave-hgs0 krb5kdc[13837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:09:09 dist-test-slave-hgs0 krb5kdc[13837](info): set up 2 sockets
Sep 27 05:09:09 dist-test-slave-hgs0 krb5kdc[13837](info): commencing operation
krb5kdc: starting...
W20260927 05:09:10.888298 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:09:11 dist-test-slave-hgs0 krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:09:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:09:11.128516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42735
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42735,127.0.97.61:34675,127.0.97.60:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:11.462961 13850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:11.463281 13850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:11.463447 13850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:11.470878 13850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:09:11.471019 13850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:11.471112 13850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:11.471215 13850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:09:11.471287 13850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:09:11.481501 13850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42735,127.0.97.61:34675,127.0.97.60:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:11.482944 13850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:11.484347 13850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:11.496457 13856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:11.496505 13855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:11.496428 13858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:11.497388 13850 server_base.cc:1034] running on GCE node
I20260927 05:09:11.498176 13850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:11.499601 13850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:11.500802 13850 hybrid_clock.cc:648] HybridClock initialized: now 1790485751500736 us; error 56 us; skew 500 ppm
I20260927 05:09:11.501188 13850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:11 dist-test-slave-hgs0 krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:11.518106 13850 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:09:11.519846 13850 webserver.cc:460] Webserver started at http://127.0.97.62:41689/ using document root <none> and password file <none>
I20260927 05:09:11.520684 13850 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:11.520823 13850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:11.521154 13850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:11.524425 13850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "1e36e24b71e2449588b838408e170a39"
format_stamp: "Formatted at 2026-09-27 05:09:11 on dist-test-slave-hgs0"
server_key: "e147eb6628b193affb9015fad27517b6"
server_key_iv: "3aac6f6a44baf6d797ba71809e6315c3"
server_key_version: "encryptionkey@0"
I20260927 05:09:11.525262 13850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "1e36e24b71e2449588b838408e170a39"
format_stamp: "Formatted at 2026-09-27 05:09:11 on dist-test-slave-hgs0"
server_key: "e147eb6628b193affb9015fad27517b6"
server_key_iv: "3aac6f6a44baf6d797ba71809e6315c3"
server_key_version: "encryptionkey@0"
I20260927 05:09:11.530978 13850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:09:11.534610 13865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:11.536204 13850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:09:11.536403 13850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "1e36e24b71e2449588b838408e170a39"
format_stamp: "Formatted at 2026-09-27 05:09:11 on dist-test-slave-hgs0"
server_key: "e147eb6628b193affb9015fad27517b6"
server_key_iv: "3aac6f6a44baf6d797ba71809e6315c3"
server_key_version: "encryptionkey@0"
I20260927 05:09:11.536626 13850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:11.556484 13850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:11.560781 13850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:11.561088 13850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:11.580158 13850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42735
I20260927 05:09:11.580200 13916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42735 every 8 connection(s)
I20260927 05:09:11.581578 13850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:09:11.585343 13917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:11.590917 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 13850
I20260927 05:09:11.591137 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:09:11.591775 388 external_mini_cluster.cc:1428] Setting key cb6dc14c029bb985d1ba3fd0f85f3d9c
I20260927 05:09:11.592342 13917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:09:11 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 05:09:11.616631 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.595749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42735 (local address 127.0.0.1:42482)
0927 05:09:11.597098 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.597141 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:11.598995 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:11.599898 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:11.599904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:11.600674 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:11.601021 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:11.601039 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:11.602606 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:11.602611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:11.603803 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:11.603816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:11.604886 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:11.604892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:11.605064 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:11.605797 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:11.605823 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:11.608855 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:11.612405 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:11.612416 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:11.612426 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:11.612868 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:11.613356 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:11.613359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:11.613362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:11.613509 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:11.614363 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:11.614373 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:11.614650 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 05:09:11.614957 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 05:09:11.615408 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"spinlock_wait_cycles":19840,"thread_start_us":837,"threads_started":1}
I20260927 05:09:11.616679 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.595748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.0.97.62:42735)
0927 05:09:11.596742 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:11.596766 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:11.597273 (+ 507us) server_negotiation.cc:407] Connection header received
0927 05:09:11.599403 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:11.599409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:11.599659 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:11.599886 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:11.601213 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:11.602433 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:11.603976 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:11.604635 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:11.604843 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:11.609125 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:11.609136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:11.609151 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:11.609188 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:11.612247 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:11.613072 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:11.613083 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:11.613088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:11.613253 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:11.613605 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:11.613608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:11.613610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:11.614243 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:11.614413 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:11.615131 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 05:09:11.615617 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":682,"thread_start_us":493,"threads_started":1}
I20260927 05:09:11.619761 13917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:09:11.621125 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.620590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50050)
0927 05:09:11.620806 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.620943 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 05:09:11.621435 13866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
I20260927 05:09:11.623040 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.622694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50062)
0927 05:09:11.622832 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.622908 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:09:11.623342 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34675: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 05:09:11.643842 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34675
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42735,127.0.97.61:34675,127.0.97.60:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:09:11.672705 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } attempt: 1
I20260927 05:09:11.673913 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.673457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50070)
0927 05:09:11.673641 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.673745 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 05:09:11.674883 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.674533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50082)
0927 05:09:11.674671 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.674740 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 05:09:11.675064 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34675: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
I20260927 05:09:11.743299 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } attempt: 2
I20260927 05:09:11.744552 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.744052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50094)
0927 05:09:11.744276 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.744389 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 05:09:11.745537 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.745187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50100)
0927 05:09:11.745330 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.745417 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 05:09:11.745718 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34675: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
I20260927 05:09:11.836994 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } attempt: 3
I20260927 05:09:11.838230 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.837763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50108)
0927 05:09:11.837976 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.838078 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 05:09:11.839452 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.839070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50124)
0927 05:09:11.839234 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.839322 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 05:09:11.839715 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34675: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
W20260927 05:09:11.975483 13924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:11.975903 13924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:11.976082 13924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 05:09:11.982998 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } attempt: 4
W20260927 05:09:11.983698 13924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:09:11.983846 13924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:11.983929 13924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:11.984037 13924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:09:11.984128 13924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:09:11.984427 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.983724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50126)
0927 05:09:11.984118 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.984240 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260927 05:09:11.985467 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:11.985100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50130)
0927 05:09:11.985252 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:11.985345 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 05:09:11.985683 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34675: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34675: connect: Connection refused (error 111)
I20260927 05:09:11.994611 13924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42735,127.0.97.61:34675,127.0.97.60:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:11.996075 13924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:11.997390 13924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:12.009575 13929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:12.009605 13932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:12.009639 13930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:12.013707 13924 server_base.cc:1034] running on GCE node
I20260927 05:09:12.014673 13924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:12.016400 13924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:12.017613 13924 hybrid_clock.cc:648] HybridClock initialized: now 1790485752017557 us; error 45 us; skew 500 ppm
I20260927 05:09:12.018045 13924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:12.021641 13924 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 05:09:12.023293 13924 webserver.cc:460] Webserver started at http://127.0.97.61:35577/ using document root <none> and password file <none>
I20260927 05:09:12.024077 13924 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:12.024194 13924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:12.024477 13924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:12.027652 13924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/data/instance:
uuid: "c24010354a4c412aacbc92ae401ca3b9"
format_stamp: "Formatted at 2026-09-27 05:09:12 on dist-test-slave-hgs0"
server_key: "9800bb005c9507ddf57859ec74b48f71"
server_key_iv: "dc84b2073857261243ed198686ad255d"
server_key_version: "encryptionkey@0"
I20260927 05:09:12.028563 13924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/wal/instance:
uuid: "c24010354a4c412aacbc92ae401ca3b9"
format_stamp: "Formatted at 2026-09-27 05:09:12 on dist-test-slave-hgs0"
server_key: "9800bb005c9507ddf57859ec74b48f71"
server_key_iv: "dc84b2073857261243ed198686ad255d"
server_key_version: "encryptionkey@0"
I20260927 05:09:12.034221 13924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 05:09:12.038213 13939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:12.039770 13924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:09:12.040030 13924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/wal
uuid: "c24010354a4c412aacbc92ae401ca3b9"
format_stamp: "Formatted at 2026-09-27 05:09:12 on dist-test-slave-hgs0"
server_key: "9800bb005c9507ddf57859ec74b48f71"
server_key_iv: "dc84b2073857261243ed198686ad255d"
server_key_version: "encryptionkey@0"
I20260927 05:09:12.040266 13924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:12.065723 13924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:12.070242 13924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:12.070540 13924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:12.089406 13924 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34675
I20260927 05:09:12.089438 13990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34675 every 8 connection(s)
I20260927 05:09:12.090758 13924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
I20260927 05:09:12.094628 13991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:12.096695 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 13924
I20260927 05:09:12.096910 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-1/wal/instance
I20260927 05:09:12.097389 388 external_mini_cluster.cc:1428] Setting key b22a912a76bf2df7df5273c65e9ea55b
I20260927 05:09:12.101183 13991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260927 05:09:12.124539 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.103932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.0.97.62:42735)
0927 05:09:12.104174 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.104181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.105569 (+ 1388us) server_negotiation.cc:407] Connection header received
0927 05:09:12.107510 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.107518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.107628 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.107766 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.109360 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.110552 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.112337 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.113192 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.113383 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.117402 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.117414 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.117422 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.117480 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.120667 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.121815 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.121821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.121823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.121964 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.122539 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.122543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.122545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.122887 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.123017 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.124104 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.124305 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:09:12.126178 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.104514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42735 (local address 127.0.0.1:42484)
0927 05:09:12.105364 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.105417 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:12.107189 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:12.108014 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:12.108023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:12.108573 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:12.109149 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.109166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.110770 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.110776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.112158 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.112173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.113504 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.113509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.113739 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.114511 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:12.114539 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:12.117075 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:12.120864 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.120875 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.120895 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.121632 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.122094 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.122102 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.122109 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.122390 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.123029 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:12.123040 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:12.123577 (+ 537us) client_negotiation.cc:769] Sending connection context
0927 05:09:12.123902 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 05:09:12.124571 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":377,"threads_started":1}
I20260927 05:09:12.128229 13991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:09:12.145047 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.129015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50134)
0927 05:09:12.129257 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.129278 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:12.129400 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:12.130531 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:12.130535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:12.130940 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:12.131083 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.131091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.133212 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.133216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.134068 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.134079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.135366 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.135371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.135513 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.136177 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:12.136195 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:12.138498 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:12.142435 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.142443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.142449 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.142817 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.143207 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.143214 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.143217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.143362 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.143956 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:12.143972 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:12.144186 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 05:09:12.144334 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 05:09:12.144637 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 05:09:12.145133 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.129256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.0.97.61:34675)
0927 05:09:12.129988 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.129999 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.130035 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:09:12.130092 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.130102 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.130366 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.130519 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.131235 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.133041 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.134215 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.135200 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.135361 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.138729 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.138748 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.138771 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.138820 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.142300 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.142979 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.142992 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.142994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.143100 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.143460 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.143467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.143470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.143819 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.144139 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.144497 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.144760 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":560,"thread_start_us":408,"threads_started":1}
I20260927 05:09:12.147699 13991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:09:12.148916 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.148429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47830)
0927 05:09:12.148605 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.148727 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:09:12.149228 13940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
I20260927 05:09:12.150997 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.150582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47834)
0927 05:09:12.150793 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.150878 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 05:09:12.151227 13991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 05:09:12.154933 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44503
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42735,127.0.97.61:34675,127.0.97.60:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:09:12.200585 13991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } attempt: 1
I20260927 05:09:12.201867 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.201385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47840)
0927 05:09:12.201582 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.201675 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 05:09:12.202881 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.202516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47842)
0927 05:09:12.202655 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.202740 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 05:09:12.203153 13991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
I20260927 05:09:12.271386 13991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } attempt: 2
I20260927 05:09:12.272749 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.272156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47844)
0927 05:09:12.272416 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.272539 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 05:09:12.273854 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.273437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47848)
0927 05:09:12.273606 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.273702 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 05:09:12.274097 13991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
I20260927 05:09:12.284938 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } attempt: 5
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:09:12.300518 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.285789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.0.97.61:34675)
0927 05:09:12.285972 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.285977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.285996 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:09:12.286183 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.286186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.286248 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.286335 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.287538 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.288366 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.290147 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.290622 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.290718 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.294851 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.294856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.294860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.294892 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.297623 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.298541 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.298547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.298549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.298686 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.299107 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.299110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.299112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.299404 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.299526 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.300107 (+ 581us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.300290 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:09:12.300724 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.285642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50150)
0927 05:09:12.285882 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.285908 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:12.286053 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:12.286402 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:12.286408 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:12.287014 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:12.287348 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.287368 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.288527 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.288533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.289986 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.290005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.290786 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.290793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.291032 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.291802 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:12.291830 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:12.294598 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:12.297804 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.297813 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.297820 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.298394 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.298809 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.298815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.298819 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.298996 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.299527 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:12.299534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:12.299724 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 05:09:12.299923 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 05:09:12.300317 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 05:09:12.301998 13917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:09:12.303205 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.302604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47856)
0927 05:09:12.302836 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.302963 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 05:09:12.304430 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.303999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47864)
0927 05:09:12.304160 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.304284 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:09:12.304761 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
I20260927 05:09:12.356046 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } attempt: 1
I20260927 05:09:12.357497 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.356770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47870)
0927 05:09:12.357065 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.357214 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 05:09:12.358677 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.358260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47874)
0927 05:09:12.358436 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.358533 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:09:12.358950 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
I20260927 05:09:12.365340 13991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } attempt: 3
I20260927 05:09:12.366595 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.366019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47890)
0927 05:09:12.366264 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.366378 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 05:09:12.367856 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.367350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47894)
0927 05:09:12.367595 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.367695 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260927 05:09:12.368109 13991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
I20260927 05:09:12.427201 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } attempt: 2
I20260927 05:09:12.428501 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.427878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47896)
0927 05:09:12.428153 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.428270 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 05:09:12.429630 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.429208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47912)
0927 05:09:12.429378 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.429481 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:09:12.429869 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
W20260927 05:09:12.495220 13998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:12.495566 13998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:12.495718 13998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:12.503032 13998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:09:12.503182 13998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:12.503288 13998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:12.503386 13998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:09:12.503450 13998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:09:12.511382 13991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } attempt: 4
I20260927 05:09:12.512696 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.512142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47924)
0927 05:09:12.512405 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.512511 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 05:09:12.513756 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.513407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47934)
0927 05:09:12.513552 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.513633 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 05:09:12.513973 13991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
I20260927 05:09:12.514413 13998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42735,127.0.97.61:34675,127.0.97.60:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:12.515707 13998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:12.517105 13998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:12.529492 14003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:12.529461 14006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:12.529531 14004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:12.530128 13998 server_base.cc:1034] running on GCE node
I20260927 05:09:12.530885 13998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:12.532425 13998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:12.533648 13998 hybrid_clock.cc:648] HybridClock initialized: now 1790485752533594 us; error 42 us; skew 500 ppm
I20260927 05:09:12.534044 13998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:09:12.536041 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } attempt: 3
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:12.537218 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.536606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47936)
0927 05:09:12.536861 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.536968 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 05:09:12.537668 13998 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 05:09:12.538379 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.537946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47950)
0927 05:09:12.538141 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.538236 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 05:09:12.538687 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44503: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44503: connect: Connection refused (error 111)
I20260927 05:09:12.539733 13998 webserver.cc:460] Webserver started at http://127.0.97.60:42433/ using document root <none> and password file <none>
I20260927 05:09:12.540555 13998 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:12.540689 13998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:12.541076 13998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:12.544368 13998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/data/instance:
uuid: "6dff7ae49817453f9336dae01fa9e990"
format_stamp: "Formatted at 2026-09-27 05:09:12 on dist-test-slave-hgs0"
server_key: "b600bb59f00c94e299dcbe5cfab249ec"
server_key_iv: "5ceffacf09e4df2a624a411434841cc3"
server_key_version: "encryptionkey@0"
I20260927 05:09:12.545249 13998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/wal/instance:
uuid: "6dff7ae49817453f9336dae01fa9e990"
format_stamp: "Formatted at 2026-09-27 05:09:12 on dist-test-slave-hgs0"
server_key: "b600bb59f00c94e299dcbe5cfab249ec"
server_key_iv: "5ceffacf09e4df2a624a411434841cc3"
server_key_version: "encryptionkey@0"
I20260927 05:09:12.550793 13998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:09:12.554389 14013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:12.555835 13998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 05:09:12.556043 13998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/wal
uuid: "6dff7ae49817453f9336dae01fa9e990"
format_stamp: "Formatted at 2026-09-27 05:09:12 on dist-test-slave-hgs0"
server_key: "b600bb59f00c94e299dcbe5cfab249ec"
server_key_iv: "5ceffacf09e4df2a624a411434841cc3"
server_key_version: "encryptionkey@0"
I20260927 05:09:12.556247 13998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:12.591212 13998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:12.595633 13998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:12.595922 13998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:12.614064 13998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44503
I20260927 05:09:12.614116 14064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44503 every 8 connection(s)
I20260927 05:09:12.615458 13998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
I20260927 05:09:12.617316 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 13998
I20260927 05:09:12.617511 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790485535376807-388-0/minicluster-data/master-2/wal/instance
I20260927 05:09:12.617987 388 external_mini_cluster.cc:1428] Setting key 9c2a9173da26bec8b3f69476d09863c6
I20260927 05:09:12.619861 14065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:12.628417 14065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:12.640838 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.619774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.0.97.60:44503)
0927 05:09:12.620627 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.620638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.620678 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:09:12.622240 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.622261 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.622682 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.623005 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.624913 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.626313 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.628146 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.628842 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.628970 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.633762 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.633769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.633779 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.633813 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.636677 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.637325 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.637331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.637342 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.637497 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.637872 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.637881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.637884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.638400 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.638694 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.639157 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.639580 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":424,"threads_started":1}
I20260927 05:09:12.646775 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.631349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.0.97.62:42735)
0927 05:09:12.631982 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.631988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.632413 (+ 425us) server_negotiation.cc:407] Connection header received
0927 05:09:12.632613 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.632619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.632692 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.632800 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.633903 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.634644 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.636062 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.636501 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.636630 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.639930 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.639979 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.639985 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.640033 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.643569 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.644638 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.644643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.644645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.644837 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.645312 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.645315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.645317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.645572 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.645698 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.646306 (+ 608us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.646487 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":277,"threads_started":1}
I20260927 05:09:12.647321 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.631297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42735 (local address 127.0.0.1:42492)
0927 05:09:12.632278 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.632319 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:12.632489 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:12.632868 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:12.632875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:12.633440 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:12.633737 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.633746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.634804 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.634808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.635861 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.635873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.636699 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.636706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.636866 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.637434 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:12.637456 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:12.639710 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:12.643709 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.643720 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.643727 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.644496 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.644964 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.644972 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.644978 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.645195 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.645693 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:12.645704 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:12.645973 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 05:09:12.646204 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 05:09:12.646627 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":589,"threads_started":1}
I20260927 05:09:12.648736 14065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:09:12.661013 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.644269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.0.97.62:42735)
0927 05:09:12.646983 (+ 2714us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.646988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.647004 (+ 16us) server_negotiation.cc:407] Connection header received
0927 05:09:12.647052 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.647059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.647169 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.647281 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.648717 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.649833 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.651051 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.651472 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.651599 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.655680 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.655687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.655691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.655726 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.658344 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.659013 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.659017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.659019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.659122 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.659511 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.659518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.659523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.659858 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.660144 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.660395 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.660605 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2623,"thread_start_us":2854,"threads_started":1}
I20260927 05:09:12.661013 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.649381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34675 (local address 127.0.0.1:50164)
0927 05:09:12.649530 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.649549 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:12.649639 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:12.649937 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:12.649941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:12.650282 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:12.650408 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.650417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.651347 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.651354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.652268 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.652279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.652962 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.652967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.653158 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.653699 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:12.653713 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:12.655743 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:12.658543 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.658546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.658549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.658877 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.659257 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.659262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.659265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.659391 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.660001 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:12.660005 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:12.660161 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 05:09:12.660320 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 05:09:12.660642 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 05:09:12.661126 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.649468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50164 (local address 127.0.97.61:34675)
0927 05:09:12.649640 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.649646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.649667 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:09:12.649746 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.649752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.649820 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.649919 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.650556 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.651211 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.652409 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.652812 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.652952 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.655917 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.655921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.655924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.655994 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.658399 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.658985 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.658993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.658999 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.659135 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.659543 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.659559 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.659561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.659857 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.660023 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.660448 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.660676 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:09:12.661984 14065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:09:12.674450 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.662640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47960)
0927 05:09:12.663039 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.663052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:12.663129 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:12.663360 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:12.663364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:12.663719 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:12.663833 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.663848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.664752 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.664760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.665579 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.665589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.666245 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.666251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.666406 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.666908 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:12.666924 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:12.669287 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:12.672170 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.672174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.672176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.672636 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.673006 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.673009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.673011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.673126 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.673736 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:12.673739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:12.673851 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 05:09:12.673976 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 05:09:12.674187 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332}
I20260927 05:09:12.674798 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.662910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.0.97.60:44503)
0927 05:09:12.663021 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.663025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.663089 (+ 64us) server_negotiation.cc:407] Connection header received
0927 05:09:12.663195 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.663198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.663246 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.663353 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.663982 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.664638 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.665713 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.666094 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.666208 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.669454 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.669458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.669462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.669487 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.672061 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.672747 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.672753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.672756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.672900 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.673222 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.673227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.673231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.673613 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.673756 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.674089 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.674365 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":31}
I20260927 05:09:12.677816 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.662688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.0.97.61:34675)
0927 05:09:12.662835 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.662840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.663294 (+ 454us) server_negotiation.cc:407] Connection header received
0927 05:09:12.663426 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.663431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.663495 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.663586 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.664794 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.665487 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.666661 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.667074 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.667207 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.671776 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.671786 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.671792 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.671831 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.675158 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.675927 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.675959 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.675963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.676127 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.676480 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.676485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.676488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.676825 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.677091 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.677253 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.677465 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 05:09:12.680611 14065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990: Bootstrap starting.
I20260927 05:09:12.682616 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:09:12.684118 14065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:12.685119 14065 log.cc:824] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:12.688560 14065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990: No bootstrap required, opened a new log
I20260927 05:09:12.702450 14065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } }
I20260927 05:09:12.702960 14065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:12.703112 14065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dff7ae49817453f9336dae01fa9e990, State: Initialized, Role: FOLLOWER
I20260927 05:09:12.704006 14065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } }
I20260927 05:09:12.708954 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } } }
I20260927 05:09:12.709511 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:09:12.710458 14065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:09:12.714944 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.687885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.0.97.62:42735)
0927 05:09:12.688168 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.688177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.692527 (+ 4350us) server_negotiation.cc:407] Connection header received
0927 05:09:12.692825 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.692842 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.692929 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.693006 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.696065 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.697528 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.700589 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.701485 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.701604 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.706431 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.706442 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.706450 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.706512 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.710757 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.711875 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.711884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.711890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.712131 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.712693 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.712701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.712707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.713188 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.713503 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.713903 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.714394 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 05:09:12.715922 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } attempt: 4
I20260927 05:09:12.721009 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.687532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.0.97.61:34675)
0927 05:09:12.687741 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.687750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.700215 (+ 12465us) server_negotiation.cc:407] Connection header received
0927 05:09:12.700727 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.700734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.700824 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.700904 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.703699 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.704907 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.707130 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.707776 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.707881 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.712558 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.712566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.712571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.712616 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.717612 (+ 4996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.718541 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.718546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.718550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.718694 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.719136 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.719141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.719144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.719542 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.719774 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.720366 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.720631 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:09:12.721699 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.692673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.0.97.60:44503)
0927 05:09:12.692860 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.692866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.703106 (+ 10240us) server_negotiation.cc:407] Connection header received
0927 05:09:12.703393 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.703400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.703481 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.703551 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.705523 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.706643 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.709044 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.709759 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.709890 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.714391 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.714399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.714405 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.714448 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.718170 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.718987 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.718993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.718997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.719184 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.719616 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.719621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.719624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.720018 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.720271 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.720816 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.721055 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:09:12.738898 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.716546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47976)
0927 05:09:12.716966 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.716991 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:12.717103 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:12.723276 (+ 6173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:12.723283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:12.723800 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:12.724085 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.724103 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.725698 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.725706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.726873 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.726887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.727757 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.727764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.728016 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.729012 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:12.729035 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:12.731582 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:12.735229 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.735235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.735239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.735769 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.736549 (+ 780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.736554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.736557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.736729 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.737560 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:12.737568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:12.737792 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 05:09:12.737956 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 05:09:12.738360 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260927 05:09:12.740255 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.717457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.0.97.60:44503)
0927 05:09:12.722898 (+ 5441us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.722906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.722923 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:09:12.722976 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.722984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.723055 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.724251 (+ 1196us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.724389 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.725538 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.727029 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.727574 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.728364 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.731739 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.731744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.731747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.731776 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.735076 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.735922 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.735929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.735955 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.736410 (+ 455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.736868 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.736873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.736875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.737367 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.739153 (+ 1786us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.739301 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.739645 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5298,"thread_start_us":13159,"threads_started":1}
I20260927 05:09:12.736636 13998 master_runner.cc:186] Error getting master registration for 127.0.97.62:42735: OK, instance_id { permanent_uuid: "1e36e24b71e2449588b838408e170a39" instance_seqno: 1790485751574447 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:09:12.745390 13917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39: Bootstrap starting.
I20260927 05:09:12.745435 13998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:09:12.748636 13917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:12.750000 13917 log.cc:824] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39: Log is configured to *not* fsync() on all Append() calls
W20260927 05:09:12.751250 14092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:09:12.751389 14092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:09:12.752933 13917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39: No bootstrap required, opened a new log
I20260927 05:09:12.760710 13917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } }
I20260927 05:09:12.761065 13917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:12.761197 13917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e36e24b71e2449588b838408e170a39, State: Initialized, Role: FOLLOWER
I20260927 05:09:12.761742 13917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } }
I20260927 05:09:12.764556 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } } }
I20260927 05:09:12.764962 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:09:12.765601 13917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:09:12.771011 14095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:09:12.771314 14095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } }
I20260927 05:09:12.774852 14095 leader_election.cc:288] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c24010354a4c412aacbc92ae401ca3b9 (127.0.97.61:34675), 6dff7ae49817453f9336dae01fa9e990 (127.0.97.60:44503)
I20260927 05:09:12.776258 14040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e36e24b71e2449588b838408e170a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dff7ae49817453f9336dae01fa9e990" is_pre_election: true
I20260927 05:09:12.776937 14040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e36e24b71e2449588b838408e170a39 in term 0.
I20260927 05:09:12.777590 13866 leader_election.cc:302] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e36e24b71e2449588b838408e170a39, 6dff7ae49817453f9336dae01fa9e990; no voters:
I20260927 05:09:12.778059 14095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:09:12.778201 14095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:09:12.778322 14095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 0 FOLLOWER]: Advancing to term 1
W20260927 05:09:12.782601 14106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:09:12.782795 14106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:09:12.783916 14095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } }
I20260927 05:09:12.786161 14040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e36e24b71e2449588b838408e170a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dff7ae49817453f9336dae01fa9e990"
I20260927 05:09:12.786409 14040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:12.790055 14040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e36e24b71e2449588b838408e170a39 in term 1.
I20260927 05:09:12.790518 13866 leader_election.cc:302] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e36e24b71e2449588b838408e170a39, 6dff7ae49817453f9336dae01fa9e990; no voters:
I20260927 05:09:12.794790 14095 leader_election.cc:288] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [CANDIDATE]: Term 1 election: Requested vote from peers c24010354a4c412aacbc92ae401ca3b9 (127.0.97.61:34675), 6dff7ae49817453f9336dae01fa9e990 (127.0.97.60:44503)
I20260927 05:09:12.795755 14095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:09:12.796489 14095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [term 1 LEADER]: Becoming Leader. State: Replica: 1e36e24b71e2449588b838408e170a39, State: Running, Role: LEADER
I20260927 05:09:12.776243 13850 master_runner.cc:186] Error getting master registration for 127.0.97.61:34675: OK, instance_id { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" instance_seqno: 1790485752084084 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:09:12.797179 13850 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:09:12.797699 14095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } }
I20260927 05:09:12.807376 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e36e24b71e2449588b838408e170a39. Latest consensus state: current_term: 1 leader_uuid: "1e36e24b71e2449588b838408e170a39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } } }
I20260927 05:09:12.807812 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:12.813186 13991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } attempt: 5
W20260927 05:09:12.814028 13871 tablet.cc:2369] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:09:12.816232 14110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:09:12.818765 14110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 05:09:12 dist-test-slave-hgs0 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:09:12.829129 14040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [term 1 FOLLOWER]: Refusing update from remote peer 1e36e24b71e2449588b838408e170a39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:09:12.830175 14108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:09:12.844594 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.813883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44503 (local address 127.0.0.1:47990)
0927 05:09:12.814292 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:12.814314 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:12.814434 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:12.815444 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:12.815450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:12.816031 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:12.816249 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.816262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.817789 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.817795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.819280 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:12.819297 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.824373 (+ 5076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.824380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:12.824602 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.825309 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:12.825335 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:12.828735 (+ 3400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:12.835132 (+ 6397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.835166 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.835170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.839061 (+ 3891us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.839989 (+ 928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:12.840169 (+ 180us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:12.840176 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:12.840458 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:12.841804 (+ 1346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:12.841809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:12.842108 (+ 299us) client_negotiation.cc:769] Sending connection context
0927 05:09:12.842327 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 05:09:12.842922 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260927 05:09:12.846175 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:12.814767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.0.97.60:44503)
0927 05:09:12.814961 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:12.814967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:12.814986 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:09:12.815051 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:12.815058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:12.815133 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:12.815239 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:12.816435 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.817603 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.823231 (+ 5628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:12.823985 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:12.824115 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:12.829047 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:12.829055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:12.829060 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:12.829099 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:12.834891 (+ 5792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.839353 (+ 4462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.839359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.839364 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.839731 (+ 367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:12.840745 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:12.840750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:12.840753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:12.841550 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:12.843426 (+ 1876us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:12.843916 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 05:09:12.845207 (+ 1291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 05:09:12.848131 14019 tablet.cc:2369] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:09:12.852336 13991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9: Bootstrap starting.
I20260927 05:09:12.856911 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e36e24b71e2449588b838408e170a39. Latest consensus state: current_term: 1 leader_uuid: "1e36e24b71e2449588b838408e170a39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } } }
I20260927 05:09:12.857226 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:09:12.857299 13991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:12.859704 13991 log.cc:824] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:12.862830 14111 mvcc.cc:205] Tried to move back new op lower bound from 7333829643574808576 to 7333829643464216576. Current Snapshot: MvccSnapshot[applied={T|T < 7333829643574808576}]
I20260927 05:09:12.877375 13991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9: No bootstrap required, opened a new log
I20260927 05:09:12.878348 14112 mvcc.cc:205] Tried to move back new op lower bound from 7333829643574808576 to 7333829643464216576. Current Snapshot: MvccSnapshot[applied={T|T < 7333829643574808576}]
I20260927 05:09:12.894155 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e36e24b71e2449588b838408e170a39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } } }
I20260927 05:09:12.894397 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:09:12.896034 13991 raft_consensus.cc:348] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } }
I20260927 05:09:12.896458 13991 raft_consensus.cc:374] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:12.896570 13991 raft_consensus.cc:729] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24010354a4c412aacbc92ae401ca3b9, State: Initialized, Role: FOLLOWER
I20260927 05:09:12.896900 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e36e24b71e2449588b838408e170a39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } } }
I20260927 05:09:12.897224 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:12.898186 13991 consensus_queue.cc:260] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } }
I20260927 05:09:12.900621 13991 sys_catalog.cc:564] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:09:12.900676 13966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e36e24b71e2449588b838408e170a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24010354a4c412aacbc92ae401ca3b9" is_pre_election: true
I20260927 05:09:12.901228 13966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e36e24b71e2449588b838408e170a39 in term 0.
I20260927 05:09:12.901604 14110 catalog_manager.cc:1347] Generated new cluster ID: 2e2055bc7706410a82a8fdb2a332e94e
I20260927 05:09:12.901695 14110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:09:12.901916 13964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:12.905478 13964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [term 1 FOLLOWER]: Refusing update from remote peer 1e36e24b71e2449588b838408e170a39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:09:12.905884 13965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e36e24b71e2449588b838408e170a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24010354a4c412aacbc92ae401ca3b9"
I20260927 05:09:12.909005 13965 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e36e24b71e2449588b838408e170a39 in term 1.
I20260927 05:09:12.910089 14108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:09:12.900002 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } } }
I20260927 05:09:12.913708 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:09:12.949628 14110 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:09:12.952108 14110 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:09:12.992377 14120 sys_catalog.cc:455] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e36e24b71e2449588b838408e170a39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } } }
I20260927 05:09:12.992820 14120 sys_catalog.cc:458] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:09:12.994093 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e36e24b71e2449588b838408e170a39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } } }
I20260927 05:09:12.994398 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:12.994686 14110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e36e24b71e2449588b838408e170a39: Generated new TSK 0
I20260927 05:09:12.995792 14110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:09:12.996770 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e36e24b71e2449588b838408e170a39. Latest consensus state: current_term: 1 leader_uuid: "1e36e24b71e2449588b838408e170a39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e36e24b71e2449588b838408e170a39" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42735 } } peers { permanent_uuid: "c24010354a4c412aacbc92ae401ca3b9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34675 } } peers { permanent_uuid: "6dff7ae49817453f9336dae01fa9e990" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44503 } } }
I20260927 05:09:12.997053 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:09:13.004482 14132 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 05:09:13.753436 14092 catalog_manager.cc:1259] Loaded cluster ID: 2e2055bc7706410a82a8fdb2a332e94e
I20260927 05:09:13.753538 14092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990: loading cluster ID for follower catalog manager: success
I20260927 05:09:13.755982 14092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990: acquiring CA information for follower catalog manager: success
I20260927 05:09:13.757711 14092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6dff7ae49817453f9336dae01fa9e990: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:09:14.008813 14132 catalog_manager.cc:1259] Loaded cluster ID: 2e2055bc7706410a82a8fdb2a332e94e
I20260927 05:09:14.008955 14132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9: loading cluster ID for follower catalog manager: success
I20260927 05:09:14.011713 14132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9: acquiring CA information for follower catalog manager: success
I20260927 05:09:14.013556 14132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c24010354a4c412aacbc92ae401ca3b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:09:14.102336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 13850
I20260927 05:09:14.103538 13912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:14.236052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 13850
W20260927 05:09:14.257364 13941 connection.cc:535] server connection from 127.0.0.1:50150 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 05:09:14.257931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 13924
I20260927 05:09:14.259073 13986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:14.371469 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 13924
I20260927 05:09:14.390328 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 13998
I20260927 05:09:14.391479 14060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:14.499161 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 13998
2026-09-27T05:09:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4737 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:09:14 dist-test-slave-hgs0 krb5kdc[14144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:09:14 dist-test-slave-hgs0 krb5kdc[14144](info): set up 2 sockets
Sep 27 05:09:14 dist-test-slave-hgs0 krb5kdc[14144](info): commencing operation
krb5kdc: starting...
W20260927 05:09:15.604568 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:09:15 dist-test-slave-hgs0 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:09:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:09:15.843364 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34771
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:16.165989 14157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:16.166308 14157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:16.166468 14157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:16.173496 14157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:09:16.173636 14157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:16.173740 14157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:16.173846 14157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:09:16.173914 14157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:09:16.183632 14157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:16.185009 14157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:16.186290 14157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:16.197942 14163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:16.198194 14165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:16.197999 14162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:16.199064 14157 server_base.cc:1034] running on GCE node
I20260927 05:09:16.199796 14157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:16.201220 14157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:16.202440 14157 hybrid_clock.cc:648] HybridClock initialized: now 1790485756202388 us; error 44 us; skew 500 ppm
I20260927 05:09:16.202831 14157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:16 dist-test-slave-hgs0 krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:16.205883 14157 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:09:16.207358 14157 webserver.cc:460] Webserver started at http://127.0.97.62:37955/ using document root <none> and password file <none>
I20260927 05:09:16.208097 14157 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:16.208204 14157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:16.208465 14157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:16.211349 14157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "d53e7b02a5a448a3a49095cc491fe516"
format_stamp: "Formatted at 2026-09-27 05:09:16 on dist-test-slave-hgs0"
server_key: "920db3d2c99cd6cce920698a982fe9cc"
server_key_iv: "0064e0c3a32bc194c135e5fcaf310ce1"
server_key_version: "encryptionkey@0"
I20260927 05:09:16.212121 14157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "d53e7b02a5a448a3a49095cc491fe516"
format_stamp: "Formatted at 2026-09-27 05:09:16 on dist-test-slave-hgs0"
server_key: "920db3d2c99cd6cce920698a982fe9cc"
server_key_iv: "0064e0c3a32bc194c135e5fcaf310ce1"
server_key_version: "encryptionkey@0"
I20260927 05:09:16.216867 14157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:09:16.220219 14172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:16.221560 14157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:09:16.221733 14157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "d53e7b02a5a448a3a49095cc491fe516"
format_stamp: "Formatted at 2026-09-27 05:09:16 on dist-test-slave-hgs0"
server_key: "920db3d2c99cd6cce920698a982fe9cc"
server_key_iv: "0064e0c3a32bc194c135e5fcaf310ce1"
server_key_version: "encryptionkey@0"
I20260927 05:09:16.221920 14157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:16.244691 14157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:16.249095 14157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:16.249400 14157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:16.267701 14157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34771
I20260927 05:09:16.267733 14223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34771 every 8 connection(s)
I20260927 05:09:16.269347 14157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:09:16.273200 14224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:16.274580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 14157
I20260927 05:09:16.274811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:09:16.275449 388 external_mini_cluster.cc:1428] Setting key b82799f8e3b6fce6c30a43a0b205c3e6
I20260927 05:09:16.285138 14224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516: Bootstrap starting.
I20260927 05:09:16.289045 14224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:16.290318 14224 log.cc:824] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:09:16 dist-test-slave-hgs0 krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:16.293378 14224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516: No bootstrap required, opened a new log
I20260927 05:09:16.299355 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:16.277487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.0.97.62:34771)
0927 05:09:16.278293 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:16.278304 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:16.278356 (+ 52us) server_negotiation.cc:407] Connection header received
0927 05:09:16.280064 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:16.280087 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:16.280469 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:16.280768 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:16.282565 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:16.283890 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:16.285824 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:16.286502 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:16.286646 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:16.292065 (+ 5419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:16.292073 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:16.292088 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:16.292125 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:16.295205 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:16.296056 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:16.296073 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:16.296081 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:16.296241 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:16.296671 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:16.296676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:16.296679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:16.297182 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:16.297530 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:16.297938 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 05:09:16.298376 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":401,"threads_started":1}
I20260927 05:09:16.301357 14224 raft_consensus.cc:348] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53e7b02a5a448a3a49095cc491fe516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34771 } }
I20260927 05:09:16.301838 14224 raft_consensus.cc:374] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:16.301944 14224 raft_consensus.cc:729] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53e7b02a5a448a3a49095cc491fe516, State: Initialized, Role: FOLLOWER
I20260927 05:09:16.302597 14224 consensus_queue.cc:260] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53e7b02a5a448a3a49095cc491fe516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34771 } }
I20260927 05:09:16.302925 14224 raft_consensus.cc:388] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:09:16.303093 14224 raft_consensus.cc:482] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:09:16.303280 14224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:16.303604 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:09:16.305540 14224 raft_consensus.cc:504] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53e7b02a5a448a3a49095cc491fe516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34771 } }
I20260927 05:09:16.306227 14224 leader_election.cc:302] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d53e7b02a5a448a3a49095cc491fe516; no voters:
I20260927 05:09:16.307092 14224 leader_election.cc:288] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:09:16.307207 14229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:09:16.307837 14229 raft_consensus.cc:686] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [term 1 LEADER]: Becoming Leader. State: Replica: d53e7b02a5a448a3a49095cc491fe516, State: Running, Role: LEADER
I20260927 05:09:16.308480 14229 consensus_queue.cc:237] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53e7b02a5a448a3a49095cc491fe516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34771 } }
I20260927 05:09:16.309394 14224 sys_catalog.cc:564] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:09:16.311299 14231 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d53e7b02a5a448a3a49095cc491fe516. Latest consensus state: current_term: 1 leader_uuid: "d53e7b02a5a448a3a49095cc491fe516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53e7b02a5a448a3a49095cc491fe516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34771 } } }
I20260927 05:09:16.311657 14231 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:16.311519 14230 sys_catalog.cc:455] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d53e7b02a5a448a3a49095cc491fe516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d53e7b02a5a448a3a49095cc491fe516" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34771 } } }
I20260927 05:09:16.311904 14230 sys_catalog.cc:458] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:16.314062 14236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:09:16.318912 14236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:09:16.327644 14236 catalog_manager.cc:1347] Generated new cluster ID: b0d01ebcf9364369a4ddaee19caf6ee1
I20260927 05:09:16.327766 14236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:09:16.347446 14236 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:09:16.348869 14236 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:09:16.359419 14236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d53e7b02a5a448a3a49095cc491fe516: Generated new TSK 0
I20260927 05:09:16.360313 14236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:09:16.416213 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 14157
I20260927 05:09:16.417443 14219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:16.513821 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 14157
2026-09-27T05:09:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2009 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:09:16 dist-test-slave-hgs0 krb5kdc[14252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:09:16 dist-test-slave-hgs0 krb5kdc[14252](info): set up 2 sockets
Sep 27 05:09:16 dist-test-slave-hgs0 krb5kdc[14252](info): commencing operation
krb5kdc: starting...
W20260927 05:09:17.680397 388 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:09:17 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:09:17.928275 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46633
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:18.260159 14265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:18.260458 14265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:18.260546 14265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:18.267825 14265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:09:18.267999 14265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:18.268069 14265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:18.268131 14265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:09:18.268173 14265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:09:18.278227 14265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:18.279659 14265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:18.280951 14265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:18.293352 14270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:18.293287 14271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:18.293314 14273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:18.294312 14265 server_base.cc:1034] running on GCE node
I20260927 05:09:18.295181 14265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:18.296677 14265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:18.297912 14265 hybrid_clock.cc:648] HybridClock initialized: now 1790485758297856 us; error 44 us; skew 500 ppm
I20260927 05:09:18.298256 14265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:18 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:18.301319 14265 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:09:18.302795 14265 webserver.cc:460] Webserver started at http://127.0.97.62:36531/ using document root <none> and password file <none>
I20260927 05:09:18.303450 14265 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:18.303547 14265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:18.303789 14265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:18.306753 14265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "4de8a83305f74ce7920488f5d3ebfe52"
format_stamp: "Formatted at 2026-09-27 05:09:18 on dist-test-slave-hgs0"
server_key: "4e83d2c68246d59ac169ec939531c45d"
server_key_iv: "80df39e200037aaf6d2f79aef863e157"
server_key_version: "encryptionkey@0"
I20260927 05:09:18.307456 14265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "4de8a83305f74ce7920488f5d3ebfe52"
format_stamp: "Formatted at 2026-09-27 05:09:18 on dist-test-slave-hgs0"
server_key: "4e83d2c68246d59ac169ec939531c45d"
server_key_iv: "80df39e200037aaf6d2f79aef863e157"
server_key_version: "encryptionkey@0"
I20260927 05:09:18.312830 14265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:09:18.316344 14280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:18.317808 14265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:09:18.317992 14265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "4de8a83305f74ce7920488f5d3ebfe52"
format_stamp: "Formatted at 2026-09-27 05:09:18 on dist-test-slave-hgs0"
server_key: "4e83d2c68246d59ac169ec939531c45d"
server_key_iv: "80df39e200037aaf6d2f79aef863e157"
server_key_version: "encryptionkey@0"
I20260927 05:09:18.318184 14265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:18.338577 14265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:18.342792 14265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:18.343109 14265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:18.362579 14265 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46633
I20260927 05:09:18.362612 14331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46633 every 8 connection(s)
I20260927 05:09:18.364048 14265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:09:18.367774 14332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:18.371179 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 14265
I20260927 05:09:18.371409 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:09:18.372120 388 external_mini_cluster.cc:1428] Setting key 64a9f8eca86cffb0eb43c6b9bf1bee77
I20260927 05:09:18.380983 14332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Bootstrap starting.
I20260927 05:09:18.384989 14332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:18.386410 14332 log.cc:824] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:09:18 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:18.389725 14332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: No bootstrap required, opened a new log
I20260927 05:09:18.396049 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:18.374083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.0.97.62:46633)
0927 05:09:18.374845 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:18.374857 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:18.374906 (+ 49us) server_negotiation.cc:407] Connection header received
0927 05:09:18.376604 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:18.376630 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:18.377100 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:18.377445 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:18.379355 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:18.380803 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:18.382675 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:18.383411 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:18.383555 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:18.388456 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:18.388464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:18.388480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:18.388519 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:18.391659 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:18.392499 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:18.392510 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:18.392517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:18.392691 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:18.393105 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:18.393110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:18.393113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:18.393669 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:18.394009 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:18.394479 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 05:09:18.394918 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":343,"threads_started":1}
I20260927 05:09:18.397976 14332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:18.398361 14332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:18.398480 14332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de8a83305f74ce7920488f5d3ebfe52, State: Initialized, Role: FOLLOWER
I20260927 05:09:18.399107 14332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:18.399339 14332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:09:18.399479 14332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:09:18.399650 14332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:18.401862 14332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:18.402369 14332 leader_election.cc:302] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de8a83305f74ce7920488f5d3ebfe52; no voters:
I20260927 05:09:18.403072 14332 leader_election.cc:288] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:09:18.403193 14337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:09:18.403867 14337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 1 LEADER]: Becoming Leader. State: Replica: 4de8a83305f74ce7920488f5d3ebfe52, State: Running, Role: LEADER
I20260927 05:09:18.404515 14337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:18.405519 14332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:09:18.408176 14339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de8a83305f74ce7920488f5d3ebfe52. Latest consensus state: current_term: 1 leader_uuid: "4de8a83305f74ce7920488f5d3ebfe52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } } }
I20260927 05:09:18.408253 14338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4de8a83305f74ce7920488f5d3ebfe52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } } }
I20260927 05:09:18.408627 14338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:18.408630 14339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:18.411293 14344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:09:18.416463 14344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:09:18.426301 14344 catalog_manager.cc:1347] Generated new cluster ID: 91c0e6c715a1458a9963834de28431f2
I20260927 05:09:18.426450 14344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:09:18.438014 14344 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:09:18.439355 14344 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:09:18.450269 14344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Generated new TSK 0
I20260927 05:09:18.451161 14344 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:09:18.512431 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46633
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:18.854749 14358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:18.855080 14358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:18.855254 14358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:18.862584 14358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:18.862757 14358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:18.862979 14358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:09:18.873479 14358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:18.874734 14358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:18.876030 14358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:18.889463 14366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:18.889521 14363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:18.889578 14364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:18.890733 14358 server_base.cc:1034] running on GCE node
I20260927 05:09:18.891388 14358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:18.892463 14358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:18.893675 14358 hybrid_clock.cc:648] HybridClock initialized: now 1790485758893619 us; error 45 us; skew 500 ppm
I20260927 05:09:18.894054 14358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:18 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:18.897868 14358 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:09:18.901073 14358 webserver.cc:460] Webserver started at http://127.0.97.1:39703/ using document root <none> and password file <none>
I20260927 05:09:18.901777 14358 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:18.901907 14358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:18.902220 14358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:18.905355 14358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "fb4f1418715a4df99a9b830ecf2b963d"
format_stamp: "Formatted at 2026-09-27 05:09:18 on dist-test-slave-hgs0"
server_key: "bb4d614e582ff30d1da0277bd33ec153"
server_key_iv: "04485e098090c4b8c0e8685c563160b9"
server_key_version: "encryptionkey@0"
I20260927 05:09:18.906188 14358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fb4f1418715a4df99a9b830ecf2b963d"
format_stamp: "Formatted at 2026-09-27 05:09:18 on dist-test-slave-hgs0"
server_key: "bb4d614e582ff30d1da0277bd33ec153"
server_key_iv: "04485e098090c4b8c0e8685c563160b9"
server_key_version: "encryptionkey@0"
I20260927 05:09:18.911586 14358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:09:18.915071 14373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:18.916523 14358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:09:18.916693 14358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "fb4f1418715a4df99a9b830ecf2b963d"
format_stamp: "Formatted at 2026-09-27 05:09:18 on dist-test-slave-hgs0"
server_key: "bb4d614e582ff30d1da0277bd33ec153"
server_key_iv: "04485e098090c4b8c0e8685c563160b9"
server_key_version: "encryptionkey@0"
I20260927 05:09:18.916908 14358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:18.931378 14358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:18.936004 14358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:18.936313 14358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:18.937304 14358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:18.938746 14358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:18.938850 14358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:18.939002 14358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:18.939074 14358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:18.973155 14358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41047
I20260927 05:09:18.973199 14485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41047 every 8 connection(s)
I20260927 05:09:18.974417 14358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:09:18.978492 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 14358
I20260927 05:09:18.978703 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:09:18.979641 388 external_mini_cluster.cc:1428] Setting key 91674b647205d927378a0d51f914eb79
Sep 27 05:09:18 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:18.998795 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:18.977062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44711 (local address 127.0.97.62:46633)
0927 05:09:18.977530 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:18.977534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:18.978527 (+ 993us) server_negotiation.cc:407] Connection header received
0927 05:09:18.981115 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:18.981124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:18.981215 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:18.981335 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:18.983402 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:18.984423 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:18.987007 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:18.987474 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:18.987606 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:18.991629 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:18.991635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:18.991638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:18.991672 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:18.994761 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:18.995819 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:18.995825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:18.995827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:18.996049 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:18.996628 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:18.996636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:18.996640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:18.996982 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:18.997162 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:18.998024 (+ 862us) server_negotiation.cc:299] Negotiation successful
0927 05:09:18.998304 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260927 05:09:19.000016 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:18.977515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:44711)
0927 05:09:18.978331 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:18.978386 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:18.980844 (+ 2458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:18.981581 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:18.981589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:18.982324 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:18.983170 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:18.983199 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:18.984603 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:18.984612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:18.986807 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:18.986825 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:18.987726 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:18.987735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:18.988127 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:18.988899 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:18.988934 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:18.991371 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:18.994994 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:18.995006 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:18.995035 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:18.995584 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:18.996190 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:18.996196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:18.996201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:18.996472 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:18.997147 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:18.997157 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:18.997578 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 05:09:18.997915 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 05:09:18.998538 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":276,"threads_started":1}
I20260927 05:09:19.001916 14486 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46633
I20260927 05:09:19.002281 14486 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:09:19.003213 14486 heartbeater.cc:511] Master 127.0.97.62:46633 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:09:19.005749 14297 ts_manager.cc:194] Registered new tserver with Master: fb4f1418715a4df99a9b830ecf2b963d (127.0.97.1:41047)
I20260927 05:09:19.007447 14297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:09:19.036002 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46633
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:19.359428 14492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:19.359736 14492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:19.359908 14492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:19.370258 14492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:19.370399 14492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:19.370570 14492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:09:19.383426 14492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:19.384689 14492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:19.385903 14492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:19.398697 14498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:19.398733 14500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:19.398670 14497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:19.399767 14492 server_base.cc:1034] running on GCE node
I20260927 05:09:19.400449 14492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:19.401329 14492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:19.402546 14492 hybrid_clock.cc:648] HybridClock initialized: now 1790485759402503 us; error 30 us; skew 500 ppm
I20260927 05:09:19.402882 14492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:19 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:19.406062 14492 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:09:19.407509 14492 webserver.cc:460] Webserver started at http://127.0.97.2:34801/ using document root <none> and password file <none>
I20260927 05:09:19.408228 14492 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:19.408332 14492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:19.408568 14492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:19.411486 14492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "c92009229c9948b7a19d815d1944d5a8"
format_stamp: "Formatted at 2026-09-27 05:09:19 on dist-test-slave-hgs0"
server_key: "cef4a6112214afe6aa918039a6a9244a"
server_key_iv: "f6d227dd903fd2a087ad812e651e00d1"
server_key_version: "encryptionkey@0"
I20260927 05:09:19.412268 14492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c92009229c9948b7a19d815d1944d5a8"
format_stamp: "Formatted at 2026-09-27 05:09:19 on dist-test-slave-hgs0"
server_key: "cef4a6112214afe6aa918039a6a9244a"
server_key_iv: "f6d227dd903fd2a087ad812e651e00d1"
server_key_version: "encryptionkey@0"
I20260927 05:09:19.417333 14492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 05:09:19.420831 14507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:19.422458 14492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:09:19.422645 14492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "c92009229c9948b7a19d815d1944d5a8"
format_stamp: "Formatted at 2026-09-27 05:09:19 on dist-test-slave-hgs0"
server_key: "cef4a6112214afe6aa918039a6a9244a"
server_key_iv: "f6d227dd903fd2a087ad812e651e00d1"
server_key_version: "encryptionkey@0"
I20260927 05:09:19.422832 14492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:19.442440 14492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:19.447146 14492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:19.447474 14492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:19.448601 14492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:19.449954 14492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:19.450032 14492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:19.450212 14492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:19.450264 14492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:19.484941 14492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32803
I20260927 05:09:19.485028 14619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32803 every 8 connection(s)
I20260927 05:09:19.486256 14492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:09:19.488385 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 14492
I20260927 05:09:19.488593 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:09:19.489082 388 external_mini_cluster.cc:1428] Setting key e4de8c3b083e85cc80bbaa138c830e60
Sep 27 05:09:19 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:19.509343 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:19.489115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48085 (local address 127.0.97.62:46633)
0927 05:09:19.489656 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:19.489662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:19.490815 (+ 1153us) server_negotiation.cc:407] Connection header received
0927 05:09:19.492818 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:19.492827 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:19.492922 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:19.493033 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:19.494554 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:19.495754 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:19.497480 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:19.498208 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:19.498318 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:19.502639 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:19.502644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:19.502647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:19.502680 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:19.506021 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:19.506883 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:19.506889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:19.506892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:19.507037 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:19.507493 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:19.507498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:19.507501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:19.507788 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:19.507945 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:19.508760 (+ 815us) server_negotiation.cc:299] Negotiation successful
0927 05:09:19.509001 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409}
I20260927 05:09:19.510414 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:19.489718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:48085)
0927 05:09:19.490630 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:19.490690 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:19.492536 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:19.493237 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:19.493246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:19.493762 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:19.494359 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:19.494377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:19.495994 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:19.495999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:19.497327 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:19.497339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:19.498475 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:19.498482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:19.498812 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:19.499569 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:19.499608 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:19.502388 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:19.506177 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:19.506188 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:19.506206 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:19.506748 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:19.507153 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:19.507157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:19.507161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:19.507368 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:19.507929 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:19.507977 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:19.508361 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 05:09:19.508645 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 05:09:19.509221 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:09:19.512209 14620 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46633
I20260927 05:09:19.512537 14620 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:19.513290 14620 heartbeater.cc:511] Master 127.0.97.62:46633 requested a full tablet report, sending...
I20260927 05:09:19.514945 14297 ts_manager.cc:194] Registered new tserver with Master: c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260927 05:09:19.515856 14297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:09:19.544261 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46633
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:19.881827 14625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:19.882103 14625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:19.882194 14625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:19.889438 14625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:19.889547 14625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:19.889657 14625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:09:19.899320 14625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:19.900561 14625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:19.901801 14625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:19.915125 14631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:19.915165 14630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:19.915203 14633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:19.916092 14625 server_base.cc:1034] running on GCE node
I20260927 05:09:19.916692 14625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:19.917687 14625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:19.919005 14625 hybrid_clock.cc:648] HybridClock initialized: now 1790485759918937 us; error 57 us; skew 500 ppm
I20260927 05:09:19.919381 14625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:19 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:19.923174 14625 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:09:19.924832 14625 webserver.cc:460] Webserver started at http://127.0.97.3:35721/ using document root <none> and password file <none>
I20260927 05:09:19.925596 14625 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:19.925740 14625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:19.926123 14625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:19.929510 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4"
format_stamp: "Formatted at 2026-09-27 05:09:19 on dist-test-slave-hgs0"
server_key: "e8c54c58f7f45df5dc5016f87f7d8da4"
server_key_iv: "220ca133719059cc597f08332988df61"
server_key_version: "encryptionkey@0"
I20260927 05:09:19.930392 14625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4"
format_stamp: "Formatted at 2026-09-27 05:09:19 on dist-test-slave-hgs0"
server_key: "e8c54c58f7f45df5dc5016f87f7d8da4"
server_key_iv: "220ca133719059cc597f08332988df61"
server_key_version: "encryptionkey@0"
I20260927 05:09:19.936105 14625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:09:19.939481 14640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:19.941002 14625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:09:19.941145 14625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4"
format_stamp: "Formatted at 2026-09-27 05:09:19 on dist-test-slave-hgs0"
server_key: "e8c54c58f7f45df5dc5016f87f7d8da4"
server_key_iv: "220ca133719059cc597f08332988df61"
server_key_version: "encryptionkey@0"
I20260927 05:09:19.941321 14625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:19.966068 14625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:19.970297 14625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:19.970539 14625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:19.971457 14625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:19.972971 14625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:19.973044 14625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:19.973165 14625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:19.973222 14625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:20.007436 14625 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34629
I20260927 05:09:20.007522 14752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34629 every 8 connection(s)
I20260927 05:09:20.008811 14625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:09:20.011643 14486 heartbeater.cc:503] Master 127.0.97.62:46633 was elected leader, sending a full tablet report...
I20260927 05:09:20.017009 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 14625
I20260927 05:09:20.017202 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:09:20.017618 388 external_mini_cluster.cc:1428] Setting key c2ef6672ddde77dff67a3cd25557a78e
Sep 27 05:09:20 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:20.029918 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.011567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46117 (local address 127.0.97.62:46633)
0927 05:09:20.012066 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.012072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.013115 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 05:09:20.015030 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.015035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.015124 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.015239 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:20.016780 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.017678 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.020044 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.020701 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.020840 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.024377 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:20.024382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:20.024386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:20.024418 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:20.027056 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:20.027805 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:20.027838 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:20.027840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:20.028025 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:20.028396 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:20.028399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:20.028401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:20.028648 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:20.028769 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.029461 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.029625 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
I20260927 05:09:20.030836 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.012168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:46117)
0927 05:09:20.012963 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:20.013000 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:20.014768 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:20.015490 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:20.015499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:20.016064 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:20.016619 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.016634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.017867 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.017872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.019789 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.019801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.020949 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.020957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.021195 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.021895 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:20.021917 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:20.024171 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:20.027191 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:20.027198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:20.027211 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:20.027681 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:20.028117 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:20.028120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:20.028123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:20.028294 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:20.028764 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:20.028770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:20.029117 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 05:09:20.029351 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 05:09:20.029728 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":365,"threads_started":1}
I20260927 05:09:20.032505 14753 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46633
I20260927 05:09:20.032886 14753 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:20.033643 14753 heartbeater.cc:511] Master 127.0.97.62:46633 requested a full tablet report, sending...
I20260927 05:09:20.034988 14297 ts_manager.cc:194] Registered new tserver with Master: fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629)
I20260927 05:09:20.035867 14297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46117
I20260927 05:09:20.046136 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:09:20.063732 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.050956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.0.97.62:46633)
0927 05:09:20.051136 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.051141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.051385 (+ 244us) server_negotiation.cc:407] Connection header received
0927 05:09:20.051520 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.051524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.051589 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.051683 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:20.053145 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.053986 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.055328 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.055807 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.055910 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.058229 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:20.058234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:20.058238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:20.058275 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:20.061341 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:20.062061 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:20.062065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:20.062068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:20.062197 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:20.062532 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:20.062536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:20.062538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:20.062878 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:20.063071 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.063306 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.063494 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:09:20.068100 14297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:09:20.074942 14297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:09:20.111820 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.087101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34629 (local address 127.0.0.1:53052)
0927 05:09:20.094096 (+ 6995us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:20.094152 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:20.094398 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:20.103752 (+ 9354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:20.103766 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:20.103811 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:20.104514 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.104528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.106486 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.106495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.109016 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.109039 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.110870 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.110876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.111021 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.111059 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 05:09:20.111170 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 05:09:20.111317 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6662,"mutex_wait_us":228,"thread_start_us":13626,"threads_started":1}
I20260927 05:09:20.111819 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.087619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.0.97.3:34629)
0927 05:09:20.102973 (+ 15354us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.102986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.103035 (+ 49us) server_negotiation.cc:407] Connection header received
0927 05:09:20.103161 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.103169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.103475 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.104483 (+ 1008us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:20.104792 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.106327 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.109235 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.110608 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.110745 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.111004 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.111272 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.111450 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15145,"thread_start_us":14914,"threads_started":1}
I20260927 05:09:20.113461 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.087113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32803 (local address 127.0.0.1:39052)
0927 05:09:20.101042 (+ 13929us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:20.101066 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:20.101206 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:20.104216 (+ 3010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:20.104225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:20.104268 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:20.104828 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.104839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.106758 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.106764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.109742 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.109760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.112611 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.112617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.112787 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.112811 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 05:09:20.112918 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 05:09:20.113050 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13780,"mutex_wait_us":72,"thread_start_us":6456,"threads_started":1}
I20260927 05:09:20.113481 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.087637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.0.97.2:32803)
0927 05:09:20.103280 (+ 15643us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.103290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.103321 (+ 31us) server_negotiation.cc:407] Connection header received
0927 05:09:20.103432 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.103441 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.103656 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.103895 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:20.105020 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.106548 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.109953 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.111494 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.111620 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.111812 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.112992 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.113174 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15460,"thread_start_us":15339,"threads_started":1}
I20260927 05:09:20.114740 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.088371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41047 (local address 127.0.0.1:37722)
0927 05:09:20.101942 (+ 13571us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:20.101972 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:20.102166 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:20.103661 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:20.103668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:20.103701 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:20.104178 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.104187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.106358 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.106366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.109747 (+ 3381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.109760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.113950 (+ 4190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.113956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.114133 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.114165 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:09:20.114253 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 05:09:20.114398 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13408,"mutex_wait_us":288,"thread_start_us":13230,"threads_started":1}
I20260927 05:09:20.115140 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.088587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.0.97.1:41047)
0927 05:09:20.101509 (+ 12922us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.103005 (+ 1496us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.103050 (+ 45us) server_negotiation.cc:407] Connection header received
0927 05:09:20.103158 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.103166 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.103425 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.103643 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:20.104397 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.106183 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.111984 (+ 5801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.113446 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.113578 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.113809 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.114348 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.114522 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12672,"thread_start_us":12524,"threads_started":1}
I20260927 05:09:20.116683 14688 tablet_service.cc:1467] Processing CreateTablet for tablet d7696ec314354236b9f7c21a8496866b (DEFAULT_TABLE table=test-table [id=a62b954c348e4453b070d7ee651e797d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:20.117293 14555 tablet_service.cc:1467] Processing CreateTablet for tablet d7696ec314354236b9f7c21a8496866b (DEFAULT_TABLE table=test-table [id=a62b954c348e4453b070d7ee651e797d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:20.118338 14688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7696ec314354236b9f7c21a8496866b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:20.119218 14555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7696ec314354236b9f7c21a8496866b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:20.119597 14421 tablet_service.cc:1467] Processing CreateTablet for tablet d7696ec314354236b9f7c21a8496866b (DEFAULT_TABLE table=test-table [id=a62b954c348e4453b070d7ee651e797d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:20.121045 14421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7696ec314354236b9f7c21a8496866b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:20.132298 14772 tablet_bootstrap.cc:492] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: Bootstrap starting.
I20260927 05:09:20.136200 14772 tablet_bootstrap.cc:654] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:20.136583 14774 tablet_bootstrap.cc:492] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: Bootstrap starting.
I20260927 05:09:20.137784 14772 log.cc:824] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:20.138422 14773 tablet_bootstrap.cc:492] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: Bootstrap starting.
I20260927 05:09:20.140357 14774 tablet_bootstrap.cc:654] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:20.141554 14773 tablet_bootstrap.cc:654] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:20.141659 14774 log.cc:824] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:20.142122 14772 tablet_bootstrap.cc:492] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: No bootstrap required, opened a new log
I20260927 05:09:20.142443 14772 ts_tablet_manager.cc:1397] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260927 05:09:20.142748 14773 log.cc:824] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:20.144860 14774 tablet_bootstrap.cc:492] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: No bootstrap required, opened a new log
I20260927 05:09:20.145246 14774 ts_tablet_manager.cc:1397] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260927 05:09:20.145483 14773 tablet_bootstrap.cc:492] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: No bootstrap required, opened a new log
I20260927 05:09:20.145725 14773 ts_tablet_manager.cc:1397] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260927 05:09:20.153101 14772 raft_consensus.cc:348] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } } peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } }
I20260927 05:09:20.153585 14772 raft_consensus.cc:374] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:20.153400 14773 raft_consensus.cc:348] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } } peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } }
I20260927 05:09:20.153687 14772 raft_consensus.cc:729] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92009229c9948b7a19d815d1944d5a8, State: Initialized, Role: FOLLOWER
I20260927 05:09:20.153820 14773 raft_consensus.cc:374] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:20.153940 14773 raft_consensus.cc:729] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc1a0d0116a4303a9cd4e13d44c6ff4, State: Initialized, Role: FOLLOWER
I20260927 05:09:20.154423 14772 consensus_queue.cc:260] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } } peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } }
I20260927 05:09:20.154532 14773 consensus_queue.cc:260] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } } peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } }
I20260927 05:09:20.155797 14620 heartbeater.cc:503] Master 127.0.97.62:46633 was elected leader, sending a full tablet report...
I20260927 05:09:20.156395 14772 ts_tablet_manager.cc:1428] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260927 05:09:20.156347 14774 raft_consensus.cc:348] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } } peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } }
I20260927 05:09:20.156828 14774 raft_consensus.cc:374] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:20.156929 14774 raft_consensus.cc:729] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4f1418715a4df99a9b830ecf2b963d, State: Initialized, Role: FOLLOWER
I20260927 05:09:20.157490 14753 heartbeater.cc:503] Master 127.0.97.62:46633 was elected leader, sending a full tablet report...
I20260927 05:09:20.157558 14774 consensus_queue.cc:260] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } } peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } }
I20260927 05:09:20.158070 14773 ts_tablet_manager.cc:1428] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260927 05:09:20.160426 14774 ts_tablet_manager.cc:1428] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260927 05:09:20.227126 14487 tablet.cc:2369] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:09:20.238427 14621 tablet.cc:2369] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:09:20.260708 14754 tablet.cc:2369] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:09:20.380400 14779 raft_consensus.cc:482] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:09:20.380738 14779 raft_consensus.cc:504] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } } peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } }
I20260927 05:09:20.382527 14779 leader_election.cc:288] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803), fb4f1418715a4df99a9b830ecf2b963d (127.0.97.1:41047)
I20260927 05:09:20.390013 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.383034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39947 (local address 127.0.97.1:41047)
0927 05:09:20.383282 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.383290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.383792 (+ 502us) server_negotiation.cc:407] Connection header received
0927 05:09:20.383902 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.383905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.384027 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.384098 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:20.384622 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.385574 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.387867 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.389134 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.389253 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.389355 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.389653 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.389772 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:09:20.390013 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.382942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55723 (local address 127.0.97.2:32803)
0927 05:09:20.383135 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.383143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.383365 (+ 222us) server_negotiation.cc:407] Connection header received
0927 05:09:20.383460 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.383465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.383553 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.383657 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:20.384222 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.385264 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.387673 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.389004 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.389138 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.389256 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.389599 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.389736 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:09:20.390077 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.382794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32803 (local address 127.0.97.3:55723)
0927 05:09:20.383123 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:20.383143 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:20.383296 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:20.383723 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:20.383730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:20.383761 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:20.384051 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.384060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.385404 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.385407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.387503 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.387518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.389178 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.389182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.389412 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.389447 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:09:20.389547 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 05:09:20.389705 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":1}
I20260927 05:09:20.390321 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.382925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41047 (local address 127.0.97.3:39947)
0927 05:09:20.383667 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:20.383683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:20.383789 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:20.384134 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:20.384140 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:20.384168 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:20.384457 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.384464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.385711 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.385716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.387720 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:20.387729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.389326 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.389331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:20.389494 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.389536 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 05:09:20.389610 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 05:09:20.389705 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":403,"threads_started":1}
I20260927 05:09:20.391181 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7696ec314354236b9f7c21a8496866b" candidate_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92009229c9948b7a19d815d1944d5a8" is_pre_election: true
I20260927 05:09:20.391373 14441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7696ec314354236b9f7c21a8496866b" candidate_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4f1418715a4df99a9b830ecf2b963d" is_pre_election: true
I20260927 05:09:20.391815 14575 raft_consensus.cc:2454] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc1a0d0116a4303a9cd4e13d44c6ff4 in term 0.
I20260927 05:09:20.391921 14441 raft_consensus.cc:2454] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc1a0d0116a4303a9cd4e13d44c6ff4 in term 0.
I20260927 05:09:20.392618 14642 leader_election.cc:302] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c92009229c9948b7a19d815d1944d5a8, fb4f1418715a4df99a9b830ecf2b963d, fbc1a0d0116a4303a9cd4e13d44c6ff4; no voters:
I20260927 05:09:20.393030 14779 raft_consensus.cc:2781] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:09:20.393186 14779 raft_consensus.cc:482] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:09:20.393298 14779 raft_consensus.cc:3037] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:20.395735 14779 raft_consensus.cc:504] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } } peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } }
I20260927 05:09:20.396397 14779 leader_election.cc:288] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [CANDIDATE]: Term 1 election: Requested vote from peers c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803), fb4f1418715a4df99a9b830ecf2b963d (127.0.97.1:41047)
I20260927 05:09:20.396872 14575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7696ec314354236b9f7c21a8496866b" candidate_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92009229c9948b7a19d815d1944d5a8"
I20260927 05:09:20.396907 14441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7696ec314354236b9f7c21a8496866b" candidate_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4f1418715a4df99a9b830ecf2b963d"
I20260927 05:09:20.397083 14575 raft_consensus.cc:3037] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:20.397104 14441 raft_consensus.cc:3037] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:20.399267 14441 raft_consensus.cc:2454] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc1a0d0116a4303a9cd4e13d44c6ff4 in term 1.
I20260927 05:09:20.399286 14575 raft_consensus.cc:2454] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc1a0d0116a4303a9cd4e13d44c6ff4 in term 1.
I20260927 05:09:20.399735 14641 leader_election.cc:302] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fb4f1418715a4df99a9b830ecf2b963d, fbc1a0d0116a4303a9cd4e13d44c6ff4; no voters:
I20260927 05:09:20.400152 14779 raft_consensus.cc:2781] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:09:20.400619 14779 raft_consensus.cc:686] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 1 LEADER]: Becoming Leader. State: Replica: fbc1a0d0116a4303a9cd4e13d44c6ff4, State: Running, Role: LEADER
I20260927 05:09:20.401326 14779 consensus_queue.cc:237] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } } peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } }
I20260927 05:09:20.406497 14296 catalog_manager.cc:5495] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } health_report { overall_health: UNKNOWN } } }
I20260927 05:09:20.476038 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.470405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.0.97.3:34629)
0927 05:09:20.470610 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.470619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.470700 (+ 81us) server_negotiation.cc:407] Connection header received
0927 05:09:20.470961 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.470965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.471054 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.471207 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:09:20.471583 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.472369 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.473837 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.474238 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.474367 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.474886 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:09:20.475396 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:09:20.475556 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.475615 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.475692 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:09:20.484644 14575 raft_consensus.cc:1261] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8 [term 1 FOLLOWER]: Refusing update from remote peer fbc1a0d0116a4303a9cd4e13d44c6ff4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:09:20.484665 14441 raft_consensus.cc:1261] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Refusing update from remote peer fbc1a0d0116a4303a9cd4e13d44c6ff4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:09:20.485767 14779 consensus_queue.cc:1035] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:09:20.486152 14782 consensus_queue.cc:1035] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:09:20.504576 14785 mvcc.cc:205] Tried to move back new op lower bound from 7333829674933841920 to 7333829674607489024. Current Snapshot: MvccSnapshot[applied={T|T < 7333829674933841920}]
I20260927 05:09:20.523977 14296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:09:20.524364 14296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:09:20.529265 14296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Sending DeleteTablet for 3 replicas of tablet d7696ec314354236b9f7c21a8496866b
I20260927 05:09:20.530638 14555 tablet_service.cc:1514] Processing DeleteTablet for tablet d7696ec314354236b9f7c21a8496866b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39052
I20260927 05:09:20.530742 14688 tablet_service.cc:1514] Processing DeleteTablet for tablet d7696ec314354236b9f7c21a8496866b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53052
I20260927 05:09:20.531661 14794 tablet_replica.cc:330] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: stopping tablet replica
I20260927 05:09:20.531903 14421 tablet_service.cc:1514] Processing DeleteTablet for tablet d7696ec314354236b9f7c21a8496866b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37722
I20260927 05:09:20.532042 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 14358
I20260927 05:09:20.532377 14794 raft_consensus.cc:2229] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:09:20.533064 14794 raft_consensus.cc:2258] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:20.535909 14794 ts_tablet_manager.cc:1905] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:09:20.542755 14796 tablet_replica.cc:330] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: stopping tablet replica
I20260927 05:09:20.543257 14796 raft_consensus.cc:2229] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:09:20.543846 14796 raft_consensus.cc:2258] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:20.545195 14794 ts_tablet_manager.cc:1918] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:09:20.545524 14794 log.cc:1196] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/d7696ec314354236b9f7c21a8496866b
I20260927 05:09:20.546145 14794 ts_tablet_manager.cc:1939] T d7696ec314354236b9f7c21a8496866b P c92009229c9948b7a19d815d1944d5a8: Deleting consensus metadata
I20260927 05:09:20.546381 14796 ts_tablet_manager.cc:1905] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:09:20.548493 14282 catalog_manager.cc:4841] TS c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803): tablet d7696ec314354236b9f7c21a8496866b (table test-table [id=a62b954c348e4453b070d7ee651e797d]) successfully deleted
I20260927 05:09:20.548836 14795 tablet_replica.cc:330] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: stopping tablet replica
I20260927 05:09:20.549229 14795 raft_consensus.cc:2229] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:09:20.549851 14795 raft_consensus.cc:2258] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:20.554957 14796 ts_tablet_manager.cc:1918] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:09:20.555150 14796 log.cc:1196] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/d7696ec314354236b9f7c21a8496866b
I20260927 05:09:20.555641 14796 ts_tablet_manager.cc:1939] T d7696ec314354236b9f7c21a8496866b P fb4f1418715a4df99a9b830ecf2b963d: Deleting consensus metadata
I20260927 05:09:20.557667 14281 catalog_manager.cc:4841] TS fb4f1418715a4df99a9b830ecf2b963d (127.0.97.1:41047): tablet d7696ec314354236b9f7c21a8496866b (table test-table [id=a62b954c348e4453b070d7ee651e797d]) successfully deleted
I20260927 05:09:20.559388 14795 ts_tablet_manager.cc:1905] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 05:09:20 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:09:20.567114 14795 ts_tablet_manager.cc:1918] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:09:20.567310 14795 log.cc:1196] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/d7696ec314354236b9f7c21a8496866b
I20260927 05:09:20.567886 14795 ts_tablet_manager.cc:1939] T d7696ec314354236b9f7c21a8496866b P fbc1a0d0116a4303a9cd4e13d44c6ff4: Deleting consensus metadata
I20260927 05:09:20.569909 14283 catalog_manager.cc:4841] TS fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629): tablet d7696ec314354236b9f7c21a8496866b (table test-table [id=a62b954c348e4453b070d7ee651e797d]) successfully deleted
I20260927 05:09:20.571848 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.533574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.0.97.1:41047)
0927 05:09:20.533797 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.533805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.551623 (+ 17818us) server_negotiation.cc:407] Connection header received
0927 05:09:20.551871 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.551878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.552004 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.552115 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:20.554188 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.555523 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.557656 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.558269 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.558377 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.564342 (+ 5965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:20.564359 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:20.564379 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:20.564434 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:20.568568 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:20.569493 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:20.569502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:20.569505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:20.569674 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:20.570150 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:20.570157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:20.570162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:20.570529 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:20.570734 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.571251 (+ 517us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.571473 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:09:20.572590 14481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:20.702881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 14358
I20260927 05:09:20.720574 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 14492
Sep 27 05:09:20 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:09:20.738511 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.721434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.0.97.2:32803)
0927 05:09:20.721677 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.721685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.721710 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:09:20.721841 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.721848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.721954 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.722083 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:20.723473 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.724851 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.726353 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.727181 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.727336 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.732437 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:20.732453 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:20.732469 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:20.732505 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:20.735408 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:20.736386 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:20.736394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:20.736397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:20.736537 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:20.737017 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:20.737021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:20.737023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:20.737395 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:20.737608 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.738057 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.738230 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:09:20.739604 14615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:20.831005 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 14492
I20260927 05:09:20.853650 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 14625
Sep 27 05:09:20 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:09:20.872587 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:20.854515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.0.97.3:34629)
0927 05:09:20.854683 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:20.854688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:20.854705 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:09:20.854941 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:20.854946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:20.855018 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:20.855099 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:20.857011 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.857864 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.859916 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:20.860514 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:20.860624 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:20.866508 (+ 5884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:20.866520 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:20.866533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:20.866573 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:20.869719 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:20.870365 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:20.870379 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:20.870382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:20.870534 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:20.870938 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:20.870943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:20.870945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:20.871306 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:20.871629 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:20.871968 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 05:09:20.872184 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:09:20.873309 14748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:20.994375 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 14625
I20260927 05:09:21.018543 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 14265
I20260927 05:09:21.019857 14327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:21.135916 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 14265
I20260927 05:09:21.157212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46633
--webserver_interface=127.0.97.62
--webserver_port=36531
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:21.482173 14802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:21.482455 14802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:21.482648 14802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:21.489848 14802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:09:21.489962 14802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:21.490049 14802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:21.490159 14802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:09:21.490221 14802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:09:21.500131 14802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:21.501505 14802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:21.502931 14802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:21.527714 14809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:21.527874 14808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:21.527714 14811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:21.528816 14802 server_base.cc:1034] running on GCE node
I20260927 05:09:21.529810 14802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:21.531497 14802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:21.532721 14802 hybrid_clock.cc:648] HybridClock initialized: now 1790485761532628 us; error 79 us; skew 500 ppm
I20260927 05:09:21.533207 14802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:21 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:21.537161 14802 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:09:21.539113 14802 webserver.cc:460] Webserver started at http://127.0.97.62:36531/ using document root <none> and password file <none>
I20260927 05:09:21.540035 14802 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:21.540182 14802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:21.546968 14802 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 05:09:21.550709 14818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:21.552570 14802 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 05:09:21.552743 14802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "4de8a83305f74ce7920488f5d3ebfe52"
format_stamp: "Formatted at 2026-09-27 05:09:18 on dist-test-slave-hgs0"
server_key: "4e83d2c68246d59ac169ec939531c45d"
server_key_iv: "80df39e200037aaf6d2f79aef863e157"
server_key_version: "encryptionkey@0"
I20260927 05:09:21.553893 14802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:21.567855 14802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:21.572414 14802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:21.572655 14802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:21.592334 14802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46633
I20260927 05:09:21.592370 14869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46633 every 8 connection(s)
I20260927 05:09:21.594002 14802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:09:21.600001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 14802
I20260927 05:09:21.601121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41047
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46633
--webserver_port=39703
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:09:21.603570 14870 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:09:21.609926 14870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Bootstrap starting.
I20260927 05:09:21.631754 14870 log.cc:824] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:21.644600 14870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:09:21.645195 14870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Bootstrap complete.
I20260927 05:09:21.655730 14870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:21.656766 14870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de8a83305f74ce7920488f5d3ebfe52, State: Initialized, Role: FOLLOWER
I20260927 05:09:21.657541 14870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:21.657902 14870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:09:21.658053 14870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:09:21.658249 14870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:09:21.662820 14870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:21.663486 14870 leader_election.cc:302] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de8a83305f74ce7920488f5d3ebfe52; no voters:
I20260927 05:09:21.664218 14870 leader_election.cc:288] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:09:21.664362 14874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:09:21.665524 14874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 2 LEADER]: Becoming Leader. State: Replica: 4de8a83305f74ce7920488f5d3ebfe52, State: Running, Role: LEADER
I20260927 05:09:21.666132 14874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:21.666692 14870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:09:21.668761 14876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de8a83305f74ce7920488f5d3ebfe52. Latest consensus state: current_term: 2 leader_uuid: "4de8a83305f74ce7920488f5d3ebfe52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } } }
I20260927 05:09:21.669095 14876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:21.668875 14875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4de8a83305f74ce7920488f5d3ebfe52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } } }
I20260927 05:09:21.669342 14875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:21.678833 14879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:09:21.690867 14879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:09:21.692812 14879 catalog_manager.cc:1259] Loaded cluster ID: 91c0e6c715a1458a9963834de28431f2
I20260927 05:09:21.692957 14879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:09:21.697449 14879 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 05:09:21.734109 14879 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 05:09:21.746758 14879 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:09:21.748705 14879 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Loaded TSK: 0
I20260927 05:09:21.750484 14879 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:09:21.966574 14872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:21.966969 14872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:21.967144 14872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:21.974895 14872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:21.975015 14872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:21.975188 14872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:09:21.985342 14872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39703
--tserver_master_addrs=127.0.97.62:46633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:21.986696 14872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:21.988181 14872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:22.004230 14897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:22.004200 14898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:22.005059 14900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:22.005111 14872 server_base.cc:1034] running on GCE node
I20260927 05:09:22.005808 14872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:22.006901 14872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:22.008137 14872 hybrid_clock.cc:648] HybridClock initialized: now 1790485762008071 us; error 57 us; skew 500 ppm
I20260927 05:09:22.008486 14872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:22 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:22.017238 14872 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:09:22.018998 14872 webserver.cc:460] Webserver started at http://127.0.97.1:39703/ using document root <none> and password file <none>
I20260927 05:09:22.019724 14872 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:22.019865 14872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:22.025457 14872 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:09:22.028327 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:22.029796 14872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:09:22.029969 14872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "fb4f1418715a4df99a9b830ecf2b963d"
format_stamp: "Formatted at 2026-09-27 05:09:18 on dist-test-slave-hgs0"
server_key: "bb4d614e582ff30d1da0277bd33ec153"
server_key_iv: "04485e098090c4b8c0e8685c563160b9"
server_key_version: "encryptionkey@0"
I20260927 05:09:22.030965 14872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:22.048413 14872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:22.052843 14872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:22.053118 14872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:22.054225 14872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:22.055675 14872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:22.055768 14872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:22.055917 14872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:22.056020 14872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 05:09:22.096354 14872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41047
I20260927 05:09:22.096393 15019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41047 every 8 connection(s)
I20260927 05:09:22.097851 14872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:09:22.099656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 14872
I20260927 05:09:22.100334 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:32803
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46633
--webserver_port=34801
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:09:22 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:22.130447 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:22.101334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57579 (local address 127.0.97.62:46633)
0927 05:09:22.108055 (+ 6721us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:22.108078 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:22.108144 (+ 66us) server_negotiation.cc:407] Connection header received
0927 05:09:22.108672 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:22.108699 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:22.109185 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:22.109642 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:22.112043 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:22.113607 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:22.116569 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:22.117345 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:22.117517 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:22.122190 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:22.122199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:22.122217 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:22.122260 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:22.125391 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:22.126238 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:22.126247 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:22.126253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:22.126425 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:22.126897 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:22.126901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:22.126903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:22.127402 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:22.127600 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:22.128489 (+ 889us) server_negotiation.cc:299] Negotiation successful
0927 05:09:22.128903 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6401,"thread_start_us":6245,"threads_started":1}
I20260927 05:09:22.130458 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:22.101612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:57579)
0927 05:09:22.102699 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:22.102754 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:22.105625 (+ 2871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:22.109828 (+ 4203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:22.109843 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:22.110731 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:22.111589 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:22.111620 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:22.113849 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:22.113859 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:22.116331 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:22.116357 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:22.117644 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:22.117654 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:22.118005 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:22.118834 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:22.118872 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:22.121853 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:22.125582 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:22.125592 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:22.125610 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:22.126071 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:22.126554 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:22.126559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:22.126563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:22.126761 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:22.127555 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:22.127563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:22.127920 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 05:09:22.128260 (+ 340us) client_negotiation.cc:240] Negotiation successful
0927 05:09:22.128850 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":457,"threads_started":1}
I20260927 05:09:22.134053 15020 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46633
I20260927 05:09:22.134384 15020 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:22.135221 15020 heartbeater.cc:511] Master 127.0.97.62:46633 requested a full tablet report, sending...
I20260927 05:09:22.137791 14835 ts_manager.cc:194] Registered new tserver with Master: fb4f1418715a4df99a9b830ecf2b963d (127.0.97.1:41047)
I20260927 05:09:22.140110 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57579
W20260927 05:09:22.456535 15024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:22.456835 15024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:22.456988 15024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:22.464385 15024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:22.464509 15024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:22.464623 15024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:09:22.474967 15024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:32803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34801
--tserver_master_addrs=127.0.97.62:46633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:22.476289 15024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:22.477526 15024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:22.490559 15032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:22.490532 15030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:22.490741 15029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:22.491326 15024 server_base.cc:1034] running on GCE node
I20260927 05:09:22.492044 15024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:22.492965 15024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:22.494172 15024 hybrid_clock.cc:648] HybridClock initialized: now 1790485762494127 us; error 43 us; skew 500 ppm
I20260927 05:09:22.494485 15024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:22 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:22.497661 15024 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:09:22.499096 15024 webserver.cc:460] Webserver started at http://127.0.97.2:34801/ using document root <none> and password file <none>
I20260927 05:09:22.499780 15024 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:22.499986 15024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:22.505489 15024 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 05:09:22.508239 15039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:22.509637 15024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:09:22.509821 15024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "c92009229c9948b7a19d815d1944d5a8"
format_stamp: "Formatted at 2026-09-27 05:09:19 on dist-test-slave-hgs0"
server_key: "cef4a6112214afe6aa918039a6a9244a"
server_key_iv: "f6d227dd903fd2a087ad812e651e00d1"
server_key_version: "encryptionkey@0"
I20260927 05:09:22.510824 15024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:22.551059 15024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:22.555436 15024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:22.555723 15024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:22.556816 15024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:22.558211 15024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:22.558331 15024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:22.558462 15024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:22.558543 15024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:22.592784 15024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32803
I20260927 05:09:22.592900 15151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32803 every 8 connection(s)
I20260927 05:09:22.594363 15024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:09:22.599928 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 15024
I20260927 05:09:22.600661 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34629
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46633
--webserver_port=35721
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:09:22 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:22.621907 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:22.597667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59437 (local address 127.0.97.62:46633)
0927 05:09:22.597906 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:22.597914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:22.599465 (+ 1551us) server_negotiation.cc:407] Connection header received
0927 05:09:22.602029 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:22.602038 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:22.602166 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:22.602304 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:22.604202 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:22.605020 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:22.606917 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:22.607484 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:22.607653 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:22.613561 (+ 5908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:22.613572 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:22.613581 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:22.613636 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:22.617950 (+ 4314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:22.618809 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:22.618818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:22.618825 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:22.619012 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:22.619486 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:22.619494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:22.619500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:22.620009 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:22.620187 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:22.621070 (+ 883us) server_negotiation.cc:299] Negotiation successful
0927 05:09:22.621405 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 05:09:22.622768 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:22.598281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:59437)
0927 05:09:22.599244 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:22.599296 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:22.601656 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:22.602608 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:22.602619 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:22.603197 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:22.603916 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:22.603998 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:22.605237 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:22.605242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:22.606708 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:22.606727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:22.607813 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:22.607823 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:22.608208 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:22.609218 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:22.609279 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:22.613225 (+ 3946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:22.618147 (+ 4922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:22.618154 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:22.618168 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:22.618642 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:22.619144 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:22.619147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:22.619149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:22.619355 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:22.620197 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:22.620204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:22.620594 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 05:09:22.620884 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 05:09:22.621368 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":408,"threads_started":1}
I20260927 05:09:22.624635 15152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46633
I20260927 05:09:22.625025 15152 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:22.625840 15152 heartbeater.cc:511] Master 127.0.97.62:46633 requested a full tablet report, sending...
I20260927 05:09:22.627491 14835 ts_manager.cc:194] Registered new tserver with Master: c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803)
I20260927 05:09:22.628484 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59437
W20260927 05:09:22.953249 15155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:22.953539 15155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:22.953706 15155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:22.961190 15155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:22.961306 15155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:22.961498 15155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:09:22.971773 15155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35721
--tserver_master_addrs=127.0.97.62:46633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:22.973256 15155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:22.974560 15155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:22.987965 15160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:22.988034 15161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:22.988065 15163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:22.989192 15155 server_base.cc:1034] running on GCE node
I20260927 05:09:22.989815 15155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:22.990852 15155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:22.992087 15155 hybrid_clock.cc:648] HybridClock initialized: now 1790485762992055 us; error 42 us; skew 500 ppm
I20260927 05:09:22.992439 15155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:22 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:22.995692 15155 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:09:22.997431 15155 webserver.cc:460] Webserver started at http://127.0.97.3:35721/ using document root <none> and password file <none>
I20260927 05:09:22.998338 15155 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:22.998509 15155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:23.004429 15155 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 05:09:23.008324 15170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:23.010360 15155 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:09:23.010536 15155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4"
format_stamp: "Formatted at 2026-09-27 05:09:19 on dist-test-slave-hgs0"
server_key: "e8c54c58f7f45df5dc5016f87f7d8da4"
server_key_iv: "220ca133719059cc597f08332988df61"
server_key_version: "encryptionkey@0"
I20260927 05:09:23.012046 15155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:23.029042 15155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:23.033286 15155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:23.033586 15155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:23.034694 15155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:23.036347 15155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:23.036474 15155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:23.036602 15155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:23.036687 15155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:23.071524 15155 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34629
I20260927 05:09:23.071575 15282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34629 every 8 connection(s)
I20260927 05:09:23.073109 15155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:09:23.081143 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 15155
Sep 27 05:09:23 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:23.094657 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.075648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38735 (local address 127.0.97.62:46633)
0927 05:09:23.075864 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.075869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.077096 (+ 1227us) server_negotiation.cc:407] Connection header received
0927 05:09:23.078914 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.078920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.079012 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.079123 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:23.080672 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.081463 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.083534 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.084050 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.084177 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.088057 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:23.088065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:23.088070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:23.088132 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:23.091361 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:23.092269 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:23.092281 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:23.092284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:23.092410 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:23.092865 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:23.092874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:23.092876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:23.093174 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:23.093341 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.094080 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.094300 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 05:09:23.095839 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.076228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:38735)
0927 05:09:23.076932 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.076970 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:23.078633 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:23.079363 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:23.079370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:23.079864 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:23.080498 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.080515 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.081612 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.081618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.083377 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.083390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.084288 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.084307 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.084539 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.085237 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:23.085261 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:23.087813 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:23.091536 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:23.091546 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:23.091564 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:23.092136 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:23.092526 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:23.092531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:23.092534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:23.092730 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:23.093309 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:23.093317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:23.093688 (+ 371us) client_negotiation.cc:769] Sending connection context
0927 05:09:23.093965 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 05:09:23.094530 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":271,"threads_started":1}
I20260927 05:09:23.096676 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.082049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.97.62:46633)
0927 05:09:23.082607 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.082613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.082890 (+ 277us) server_negotiation.cc:407] Connection header received
0927 05:09:23.083090 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.083095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.083154 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.083263 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:23.084761 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.085697 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.087370 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.087817 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.087983 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.090634 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:23.090639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:23.090642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:23.090675 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:23.093607 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:23.094317 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:23.094323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:23.094327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:23.094449 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:23.094914 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:23.094919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:23.094921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:23.095210 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:23.095471 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.095888 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.096265 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"spinlock_wait_cycles":1792,"thread_start_us":278,"threads_started":1}
I20260927 05:09:23.101055 15283 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46633
I20260927 05:09:23.101411 15283 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:23.102173 15283 heartbeater.cc:511] Master 127.0.97.62:46633 requested a full tablet report, sending...
I20260927 05:09:23.103678 14835 ts_manager.cc:194] Registered new tserver with Master: fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629)
I20260927 05:09:23.104604 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38735
I20260927 05:09:23.112042 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:09:23.119329 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.112952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.0.97.62:46633)
0927 05:09:23.113159 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.113163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.113604 (+ 441us) server_negotiation.cc:407] Connection header received
0927 05:09:23.113868 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.113872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.114028 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.114180 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:09:23.114697 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.115432 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.117069 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.117566 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.117669 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.118243 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:09:23.118784 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:09:23.118908 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.119024 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.119103 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 05:09:23.120366 14835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:09:23.127918 14835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:09:23.143925 15020 heartbeater.cc:503] Master 127.0.97.62:46633 was elected leader, sending a full tablet report...
I20260927 05:09:23.166785 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.142437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41047 (local address 127.0.0.1:37744)
0927 05:09:23.155836 (+ 13399us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.155857 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:23.156046 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:23.157093 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:23.157098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:23.157135 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:23.157646 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.157665 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.160327 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.160333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.162715 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.162735 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.165547 (+ 2812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.165557 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.165811 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.165879 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 05:09:23.166052 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 05:09:23.166300 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13259,"mutex_wait_us":2,"thread_start_us":12033,"threads_started":1}
I20260927 05:09:23.168001 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.145276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.0.97.1:41047)
0927 05:09:23.155352 (+ 10076us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.155363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.156218 (+ 855us) server_negotiation.cc:407] Connection header received
0927 05:09:23.156350 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.156359 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.156606 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.156807 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:23.157912 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.160132 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.162977 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.164935 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.165083 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.165381 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.167358 (+ 1977us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.167572 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9895,"thread_start_us":9739,"threads_started":1}
I20260927 05:09:23.168488 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.144642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.0.97.2:32803)
0927 05:09:23.154836 (+ 10194us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.154846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.156591 (+ 1745us) server_negotiation.cc:407] Connection header received
0927 05:09:23.156701 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.156712 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.157020 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.157209 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:23.158716 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.160360 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.163751 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.165509 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.165649 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.165872 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.167877 (+ 2005us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.168126 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10017,"thread_start_us":9864,"threads_started":1}
I20260927 05:09:23.167725 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.142987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32803 (local address 127.0.0.1:39064)
0927 05:09:23.156332 (+ 13345us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.156351 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:23.156465 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:23.157464 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:23.157471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:23.157499 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:23.158516 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.158532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.160531 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.160537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.163525 (+ 2988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.163543 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.166060 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.166067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.166279 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.166312 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:09:23.167120 (+ 808us) client_negotiation.cc:240] Negotiation successful
0927 05:09:23.167310 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13216,"mutex_wait_us":1,"thread_start_us":12528,"threads_started":1}
I20260927 05:09:23.169946 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.142175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34629 (local address 127.0.0.1:53072)
0927 05:09:23.154963 (+ 12788us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.155042 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:23.155318 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:23.156971 (+ 1653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:23.156980 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:23.157008 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:23.157516 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.157536 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.159883 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.159889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.164639 (+ 4750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.164661 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.169043 (+ 4382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.169052 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.169270 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.169311 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 05:09:23.169433 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 05:09:23.169584 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12460,"thread_start_us":13832,"threads_started":1}
I20260927 05:09:23.169976 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.142445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.0.97.3:34629)
0927 05:09:23.154205 (+ 11760us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.154219 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.155881 (+ 1662us) server_negotiation.cc:407] Connection header received
0927 05:09:23.156071 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.156079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.156396 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.156610 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:23.157778 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.159667 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.165181 (+ 5514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.168380 (+ 3199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.168560 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.168832 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.169510 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.169668 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11510,"thread_start_us":11321,"threads_started":1}
I20260927 05:09:23.174772 15087 tablet_service.cc:1467] Processing CreateTablet for tablet 38b8ecdffd5f4f23972d448da6d177ff (DEFAULT_TABLE table=test-table [id=cf855f7388a74bbaa39052652c29afcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:23.174829 14955 tablet_service.cc:1467] Processing CreateTablet for tablet 38b8ecdffd5f4f23972d448da6d177ff (DEFAULT_TABLE table=test-table [id=cf855f7388a74bbaa39052652c29afcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:23.175434 15218 tablet_service.cc:1467] Processing CreateTablet for tablet 38b8ecdffd5f4f23972d448da6d177ff (DEFAULT_TABLE table=test-table [id=cf855f7388a74bbaa39052652c29afcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:23.176172 15087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b8ecdffd5f4f23972d448da6d177ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:23.176331 14955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b8ecdffd5f4f23972d448da6d177ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:23.176752 15218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b8ecdffd5f4f23972d448da6d177ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:23.190908 15299 tablet_bootstrap.cc:492] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: Bootstrap starting.
I20260927 05:09:23.193404 15301 tablet_bootstrap.cc:492] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: Bootstrap starting.
I20260927 05:09:23.196254 15299 tablet_bootstrap.cc:654] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:23.196563 15300 tablet_bootstrap.cc:492] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Bootstrap starting.
I20260927 05:09:23.197355 15301 tablet_bootstrap.cc:654] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:23.198104 15299 log.cc:824] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:23.198829 15301 log.cc:824] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:23.200227 15300 tablet_bootstrap.cc:654] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:23.201502 15299 tablet_bootstrap.cc:492] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: No bootstrap required, opened a new log
I20260927 05:09:23.201543 15300 log.cc:824] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:23.201830 15299 ts_tablet_manager.cc:1397] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 05:09:23.202316 15301 tablet_bootstrap.cc:492] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: No bootstrap required, opened a new log
I20260927 05:09:23.202647 15301 ts_tablet_manager.cc:1397] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 05:09:23.206435 15300 tablet_bootstrap.cc:492] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: No bootstrap required, opened a new log
I20260927 05:09:23.206748 15300 ts_tablet_manager.cc:1397] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260927 05:09:23.209955 15299 raft_consensus.cc:348] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:23.210353 15299 raft_consensus.cc:374] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:23.210467 15299 raft_consensus.cc:729] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92009229c9948b7a19d815d1944d5a8, State: Initialized, Role: FOLLOWER
I20260927 05:09:23.211113 15299 consensus_queue.cc:260] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:23.212347 15152 heartbeater.cc:503] Master 127.0.97.62:46633 was elected leader, sending a full tablet report...
I20260927 05:09:23.212873 15299 ts_tablet_manager.cc:1428] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 05:09:23.213426 15301 raft_consensus.cc:348] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:23.213898 15301 raft_consensus.cc:374] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:23.214016 15301 raft_consensus.cc:729] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc1a0d0116a4303a9cd4e13d44c6ff4, State: Initialized, Role: FOLLOWER
I20260927 05:09:23.214776 15301 consensus_queue.cc:260] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:23.216813 15301 ts_tablet_manager.cc:1428] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260927 05:09:23.216837 15283 heartbeater.cc:503] Master 127.0.97.62:46633 was elected leader, sending a full tablet report...
I20260927 05:09:23.219062 15300 raft_consensus.cc:348] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:23.219617 15300 raft_consensus.cc:374] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:23.219760 15300 raft_consensus.cc:729] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4f1418715a4df99a9b830ecf2b963d, State: Initialized, Role: FOLLOWER
I20260927 05:09:23.220502 15300 consensus_queue.cc:260] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:23.222255 15300 ts_tablet_manager.cc:1428] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 05:09:23.315162 15307 raft_consensus.cc:482] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:09:23.315598 15307 raft_consensus.cc:504] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:23.317718 15307 leader_election.cc:288] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629), c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803)
I20260927 05:09:23.324622 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.318558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46095 (local address 127.0.97.3:34629)
0927 05:09:23.318758 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.318767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.319454 (+ 687us) server_negotiation.cc:407] Connection header received
0927 05:09:23.319606 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.319610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.319669 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.319744 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:23.320414 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.321124 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.322904 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.323689 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.323827 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.323945 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.324272 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.324406 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:09:23.324638 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.318128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54867 (local address 127.0.97.2:32803)
0927 05:09:23.318616 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.318625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.318820 (+ 195us) server_negotiation.cc:407] Connection header received
0927 05:09:23.319145 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.319152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.319236 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.319316 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:23.320141 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.320907 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.322783 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.323564 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.323728 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.323834 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.324222 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.324379 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322}
W20260927 05:09:23.324772 15284 tablet.cc:2369] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:09:23.325405 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.318052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34629 (local address 127.0.97.1:46095)
0927 05:09:23.319348 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.319367 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:23.319486 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:23.319803 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:23.319808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:23.319841 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:23.320248 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.320258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.321276 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.321282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.322746 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.322759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.323959 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.323967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.324102 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.324125 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:09:23.324226 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 05:09:23.324362 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"thread_start_us":1009,"threads_started":1}
I20260927 05:09:23.325403 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.317939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32803 (local address 127.0.97.1:54867)
0927 05:09:23.318717 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.318737 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:23.318942 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:23.319527 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:23.319533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:23.319560 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:23.320000 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.320017 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.321052 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.321060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.322603 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:23.322618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.323799 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.323805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:23.324057 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.324083 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 05:09:23.324184 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 05:09:23.324324 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":821,"threads_started":1}
I20260927 05:09:23.326413 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b8ecdffd5f4f23972d448da6d177ff" candidate_uuid: "fb4f1418715a4df99a9b830ecf2b963d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92009229c9948b7a19d815d1944d5a8" is_pre_election: true
I20260927 05:09:23.326475 15238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b8ecdffd5f4f23972d448da6d177ff" candidate_uuid: "fb4f1418715a4df99a9b830ecf2b963d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" is_pre_election: true
I20260927 05:09:23.326939 15238 raft_consensus.cc:2454] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4f1418715a4df99a9b830ecf2b963d in term 0.
I20260927 05:09:23.326933 15107 raft_consensus.cc:2454] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4f1418715a4df99a9b830ecf2b963d in term 0.
I20260927 05:09:23.327605 14909 leader_election.cc:302] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c92009229c9948b7a19d815d1944d5a8, fb4f1418715a4df99a9b830ecf2b963d, fbc1a0d0116a4303a9cd4e13d44c6ff4; no voters:
I20260927 05:09:23.328042 15307 raft_consensus.cc:2781] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:09:23.328198 15307 raft_consensus.cc:482] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:09:23.328313 15307 raft_consensus.cc:3037] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:23.330487 15307 raft_consensus.cc:504] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:23.331084 15307 leader_election.cc:288] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 1 election: Requested vote from peers fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629), c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803)
I20260927 05:09:23.331594 15107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b8ecdffd5f4f23972d448da6d177ff" candidate_uuid: "fb4f1418715a4df99a9b830ecf2b963d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92009229c9948b7a19d815d1944d5a8"
I20260927 05:09:23.331594 15238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b8ecdffd5f4f23972d448da6d177ff" candidate_uuid: "fb4f1418715a4df99a9b830ecf2b963d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4"
I20260927 05:09:23.331820 15107 raft_consensus.cc:3037] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:23.331815 15238 raft_consensus.cc:3037] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:23.335106 15238 raft_consensus.cc:2454] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4f1418715a4df99a9b830ecf2b963d in term 1.
I20260927 05:09:23.335106 15107 raft_consensus.cc:2454] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4f1418715a4df99a9b830ecf2b963d in term 1.
I20260927 05:09:23.335552 14910 leader_election.cc:302] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c92009229c9948b7a19d815d1944d5a8, fb4f1418715a4df99a9b830ecf2b963d, fbc1a0d0116a4303a9cd4e13d44c6ff4; no voters:
I20260927 05:09:23.335959 15307 raft_consensus.cc:2781] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:09:23.336421 15307 raft_consensus.cc:686] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 LEADER]: Becoming Leader. State: Replica: fb4f1418715a4df99a9b830ecf2b963d, State: Running, Role: LEADER
I20260927 05:09:23.336969 15307 consensus_queue.cc:237] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:23.341626 14835 catalog_manager.cc:5495] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4f1418715a4df99a9b830ecf2b963d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fb4f1418715a4df99a9b830ecf2b963d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } health_report { overall_health: UNKNOWN } } }
W20260927 05:09:23.346546 15153 tablet.cc:2369] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:09:23.350258 15021 tablet.cc:2369] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:09:23.369850 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.363870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.0.97.1:41047)
0927 05:09:23.364136 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.364144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.364244 (+ 100us) server_negotiation.cc:407] Connection header received
0927 05:09:23.364515 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.364520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.364610 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.364776 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:09:23.365168 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.365886 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.367501 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.368046 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.368198 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.368714 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:09:23.369256 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:09:23.369439 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.369489 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.369575 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 05:09:23.378587 15238 raft_consensus.cc:1261] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 1 FOLLOWER]: Refusing update from remote peer fb4f1418715a4df99a9b830ecf2b963d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:09:23.378585 15107 raft_consensus.cc:1261] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8 [term 1 FOLLOWER]: Refusing update from remote peer fb4f1418715a4df99a9b830ecf2b963d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:09:23.379596 15310 consensus_queue.cc:1035] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:09:23.379982 15307 consensus_queue.cc:1035] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:09:23.397867 15315 mvcc.cc:205] Tried to move back new op lower bound from 7333829686788767744 to 7333829686632157184. Current Snapshot: MvccSnapshot[applied={T|T < 7333829686788767744}]
I20260927 05:09:23.409307 14835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:09:23.409554 14835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:09:23.413298 14835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Sending DeleteTablet for 3 replicas of tablet 38b8ecdffd5f4f23972d448da6d177ff
I20260927 05:09:23.414772 15087 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b8ecdffd5f4f23972d448da6d177ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39064
I20260927 05:09:23.414762 15218 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b8ecdffd5f4f23972d448da6d177ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53072
I20260927 05:09:23.415441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 14872
I20260927 05:09:23.415597 14955 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b8ecdffd5f4f23972d448da6d177ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37744
I20260927 05:09:23.423877 15325 tablet_replica.cc:330] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: stopping tablet replica
I20260927 05:09:23.425467 15325 raft_consensus.cc:2229] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:09:23.427325 15325 raft_consensus.cc:2258] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:23.429376 15323 tablet_replica.cc:330] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: stopping tablet replica
I20260927 05:09:23.429868 15323 raft_consensus.cc:2229] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:09:23.430433 15323 raft_consensus.cc:2258] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:23.430835 15325 ts_tablet_manager.cc:1905] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:09:23.431552 15324 tablet_replica.cc:330] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: stopping tablet replica
I20260927 05:09:23.432094 15324 raft_consensus.cc:2229] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:09:23.432593 15324 raft_consensus.cc:2258] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:23.433411 15298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.416416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.0.97.1:41047)
0927 05:09:23.416608 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.416614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.416707 (+ 93us) server_negotiation.cc:407] Connection header received
0927 05:09:23.416925 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.416933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.417033 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.417156 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:23.419107 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.420092 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.421944 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.422485 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.422647 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.426317 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:23.426337 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:23.426353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:23.426392 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:23.429883 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:23.430914 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:23.430926 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:23.430932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:23.431107 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:23.431560 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:23.431566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:23.431569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:23.432046 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:23.432386 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.432729 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.433010 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:09:23.434106 15015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:23.434892 15324 ts_tablet_manager.cc:1905] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:09:23.434892 15323 ts_tablet_manager.cc:1905] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:09:23.441999 15324 ts_tablet_manager.cc:1918] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:09:23.442175 15324 log.cc:1196] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/38b8ecdffd5f4f23972d448da6d177ff
I20260927 05:09:23.442785 15324 ts_tablet_manager.cc:1939] T 38b8ecdffd5f4f23972d448da6d177ff P fbc1a0d0116a4303a9cd4e13d44c6ff4: Deleting consensus metadata
I20260927 05:09:23.442867 15323 ts_tablet_manager.cc:1918] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:09:23.443024 15323 log.cc:1196] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/38b8ecdffd5f4f23972d448da6d177ff
I20260927 05:09:23.443555 15323 ts_tablet_manager.cc:1939] T 38b8ecdffd5f4f23972d448da6d177ff P c92009229c9948b7a19d815d1944d5a8: Deleting consensus metadata
I20260927 05:09:23.444882 14821 catalog_manager.cc:4841] TS fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629): tablet 38b8ecdffd5f4f23972d448da6d177ff (table test-table [id=cf855f7388a74bbaa39052652c29afcd]) successfully deleted
I20260927 05:09:23.445226 14820 catalog_manager.cc:4841] TS c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803): tablet 38b8ecdffd5f4f23972d448da6d177ff (table test-table [id=cf855f7388a74bbaa39052652c29afcd]) successfully deleted
I20260927 05:09:23.532593 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 14872
I20260927 05:09:23.552052 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 15024
W20260927 05:09:23.552160 14819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41047 (error 108)
I20260927 05:09:23.553995 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.553561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41047 (local address 127.0.0.1:37766)
0927 05:09:23.553759 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.553853 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:09:23.554257 14819 catalog_manager.cc:4584] TS fb4f1418715a4df99a9b830ecf2b963d (127.0.97.1:41047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38b8ecdffd5f4f23972d448da6d177ff: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41047: connect: Connection refused (error 111)
I20260927 05:09:23.567152 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.552896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.0.97.2:32803)
0927 05:09:23.553073 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.553079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.553187 (+ 108us) server_negotiation.cc:407] Connection header received
0927 05:09:23.553404 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.553410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.553499 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.553610 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:23.555065 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.555799 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.557213 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.557647 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.557805 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.560617 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:23.560631 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:23.560654 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:23.560701 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:23.563676 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:23.564866 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:23.564873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:23.564876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:23.565011 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:23.565482 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:23.565486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:23.565489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:23.565954 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:23.566174 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.566560 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.566735 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:09:23.567915 15147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:23.608162 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.607543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41047 (local address 127.0.0.1:37780)
0927 05:09:23.607784 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.607888 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 05:09:23.609409 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.609026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41047 (local address 127.0.0.1:37788)
0927 05:09:23.609186 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.609279 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 05:09:23.660287 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 15024
I20260927 05:09:23.670173 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.669635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41047 (local address 127.0.0.1:37798)
0927 05:09:23.669913 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.670009 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 05:09:23.671474 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.671011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41047 (local address 127.0.0.1:37814)
0927 05:09:23.671190 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.671334 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 05:09:23.679119 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 15155
I20260927 05:09:23.695152 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.680135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.0.97.3:34629)
0927 05:09:23.680415 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:23.680422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:23.680448 (+ 26us) server_negotiation.cc:407] Connection header received
0927 05:09:23.680600 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:23.680606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:23.680694 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:23.680831 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:23.682469 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.683321 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.684897 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:23.685434 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:23.685582 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:23.688439 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:23.688456 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:23.688473 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:23.688517 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:23.691612 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:23.692811 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:23.692829 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:23.692832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:23.692986 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:23.693431 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:23.693436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:23.693438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:23.693846 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:23.694178 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:23.694525 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 05:09:23.694770 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:09:23.695825 15278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:23.752166 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.751525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41047 (local address 127.0.0.1:37818)
0927 05:09:23.751766 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.751897 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 05:09:23.753294 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:23.752874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41047 (local address 127.0.0.1:37832)
0927 05:09:23.753089 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:23.753175 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 05:09:23.791615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 15155
I20260927 05:09:23.810374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 14802
I20260927 05:09:23.811789 14865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:23.934233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 14802
I20260927 05:09:23.957892 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46633
--webserver_interface=127.0.97.62
--webserver_port=36531
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:24.323450 15331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:24.323825 15331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:24.324004 15331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:24.331440 15331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:09:24.331605 15331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:24.331717 15331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:24.331833 15331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:09:24.331910 15331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:09:24.342187 15331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:24.343770 15331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:24.345178 15331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:24.357209 15337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:24.357146 15339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:24.357450 15336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:24.358261 15331 server_base.cc:1034] running on GCE node
I20260927 05:09:24.359040 15331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:24.360610 15331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:24.361858 15331 hybrid_clock.cc:648] HybridClock initialized: now 1790485764361797 us; error 45 us; skew 500 ppm
I20260927 05:09:24.362293 15331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:24 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:24.365581 15331 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:09:24.367249 15331 webserver.cc:460] Webserver started at http://127.0.97.62:36531/ using document root <none> and password file <none>
I20260927 05:09:24.367985 15331 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:24.368108 15331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:24.373658 15331 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:09:24.376587 15346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:24.377987 15331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:09:24.378180 15331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "4de8a83305f74ce7920488f5d3ebfe52"
format_stamp: "Formatted at 2026-09-27 05:09:18 on dist-test-slave-hgs0"
server_key: "4e83d2c68246d59ac169ec939531c45d"
server_key_iv: "80df39e200037aaf6d2f79aef863e157"
server_key_version: "encryptionkey@0"
I20260927 05:09:24.379207 15331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:24.391072 15331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:24.395511 15331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:24.395818 15331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:24.417016 15331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46633
I20260927 05:09:24.417079 15397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46633 every 8 connection(s)
I20260927 05:09:24.418996 15331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:09:24.425391 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 15331
I20260927 05:09:24.426255 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41047
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46633
--webserver_port=39703
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:09:24.427119 15398 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:09:24.430684 15398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Bootstrap starting.
I20260927 05:09:24.462634 15398 log.cc:824] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:24.484647 15398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:09:24.485262 15398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52: Bootstrap complete.
I20260927 05:09:24.495059 15398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:24.496119 15398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4de8a83305f74ce7920488f5d3ebfe52, State: Initialized, Role: FOLLOWER
I20260927 05:09:24.496985 15398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:24.497337 15398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:09:24.497502 15398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:09:24.497721 15398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 2 FOLLOWER]: Advancing to term 3
I20260927 05:09:24.502262 15398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:24.502887 15398 leader_election.cc:302] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4de8a83305f74ce7920488f5d3ebfe52; no voters:
I20260927 05:09:24.503691 15398 leader_election.cc:288] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 05:09:24.503862 15402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 05:09:24.505302 15402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [term 3 LEADER]: Becoming Leader. State: Replica: 4de8a83305f74ce7920488f5d3ebfe52, State: Running, Role: LEADER
I20260927 05:09:24.505914 15402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } }
I20260927 05:09:24.506412 15398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:09:24.510174 15404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4de8a83305f74ce7920488f5d3ebfe52. Latest consensus state: current_term: 3 leader_uuid: "4de8a83305f74ce7920488f5d3ebfe52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } } }
I20260927 05:09:24.510632 15404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:24.516490 15403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4de8a83305f74ce7920488f5d3ebfe52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4de8a83305f74ce7920488f5d3ebfe52" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46633 } } }
I20260927 05:09:24.516860 15403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4de8a83305f74ce7920488f5d3ebfe52 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:24.524240 15409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:09:24.536912 15409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:09:24.539033 15409 catalog_manager.cc:1259] Loaded cluster ID: 91c0e6c715a1458a9963834de28431f2
I20260927 05:09:24.539184 15409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 05:09:24.542101 15409 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ffb9f5a99ad google::LogMessage::Fail() at ??:0
@ 0x7ffb9f5ac88a google::LogMessage::SendToLog() at ??:0
@ 0x7ffb9f5a94de google::LogMessage::Flush() at ??:0
@ 0x7ffb9f5aaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260927 05:09:24.552825 15419 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7ffbb4fdc512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ffbb4fd9976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ffbb5061610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ffbb5061439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ffba21f5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ffba21f891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ffba21f8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ffba21d3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffba54ad6db start_thread at ??:0
@ 0x7ffb9e44361f clone at ??:0
W20260927 05:09:24.850736 15400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:24.851086 15400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:24.851259 15400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:24.858788 15400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:24.858954 15400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:24.859153 15400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:09:24.869462 15400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=39703
--tserver_master_addrs=127.0.97.62:46633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:24.870901 15400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:24.872323 15400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:24.886404 15439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:24.886461 15442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:24.886545 15440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:24.887295 15400 server_base.cc:1034] running on GCE node
I20260927 05:09:24.888106 15400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:24.889223 15400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:24.890496 15400 hybrid_clock.cc:648] HybridClock initialized: now 1790485764890437 us; error 48 us; skew 500 ppm
I20260927 05:09:24.890991 15400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:24 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:24.894382 15400 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:09:24.896035 15400 webserver.cc:460] Webserver started at http://127.0.97.1:39703/ using document root <none> and password file <none>
I20260927 05:09:24.896708 15400 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:24.896818 15400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:24.902407 15400 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:09:24.905684 15449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:24.907409 15400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:09:24.907647 15400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "fb4f1418715a4df99a9b830ecf2b963d"
format_stamp: "Formatted at 2026-09-27 05:09:18 on dist-test-slave-hgs0"
server_key: "bb4d614e582ff30d1da0277bd33ec153"
server_key_iv: "04485e098090c4b8c0e8685c563160b9"
server_key_version: "encryptionkey@0"
I20260927 05:09:24.908923 15400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:24.970485 15400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:24.975430 15400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:24.975745 15400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:24.976953 15400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:24.979040 15456 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 05:09:24.985499 15400 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 05:09:24.985687 15400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260927 05:09:24.985846 15400 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 05:09:24.989611 15400 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 05:09:24.989741 15400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.003s
I20260927 05:09:24.990132 15456 tablet_bootstrap.cc:492] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Bootstrap starting.
I20260927 05:09:25.000044 15456 log.cc:824] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:25.017021 15456 tablet_bootstrap.cc:492] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:09:25.017905 15456 tablet_bootstrap.cc:492] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Bootstrap complete.
I20260927 05:09:25.018523 15456 ts_tablet_manager.cc:1397] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20260927 05:09:25.028643 15456 raft_consensus.cc:348] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:25.030755 15456 raft_consensus.cc:729] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4f1418715a4df99a9b830ecf2b963d, State: Initialized, Role: FOLLOWER
I20260927 05:09:25.031747 15456 consensus_queue.cc:260] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:25.034181 15456 ts_tablet_manager.cc:1428] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 05:09:25.046206 15400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41047
I20260927 05:09:25.046258 15564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41047 every 8 connection(s)
I20260927 05:09:25.047722 15400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:09:25.053388 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:25.051174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:55361)
0927 05:09:25.052187 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:25.052418 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":359,"threads_started":1}
W20260927 05:09:25.054200 15565 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46633 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46633: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
I20260927 05:09:25.055187 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:25.054829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:48665)
0927 05:09:25.054992 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:25.055067 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 05:09:25.056208 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:25.055809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:52607)
0927 05:09:25.056009 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:25.056084 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:09:25.056468 15565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:09:25.058195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 15400
I20260927 05:09:25.058813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:32803
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46633
--webserver_port=34801
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:25.413872 15568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:25.414222 15568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:25.414445 15568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:25.421809 15568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:25.421993 15568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:25.422211 15568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:09:25.432322 15568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:32803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34801
--tserver_master_addrs=127.0.97.62:46633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:25.433804 15568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:25.435134 15568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:25.448346 15573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:25.448378 15574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:25.448577 15576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:25.449726 15568 server_base.cc:1034] running on GCE node
I20260927 05:09:25.450512 15568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:25.451583 15568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:25.452804 15568 hybrid_clock.cc:648] HybridClock initialized: now 1790485765452740 us; error 44 us; skew 500 ppm
I20260927 05:09:25.453178 15568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:25 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:25.456779 15568 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:09:25.458389 15568 webserver.cc:460] Webserver started at http://127.0.97.2:34801/ using document root <none> and password file <none>
I20260927 05:09:25.459141 15568 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:25.459295 15568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:25.465247 15568 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:09:25.469051 15583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:25.470553 15568 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:09:25.470734 15568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "c92009229c9948b7a19d815d1944d5a8"
format_stamp: "Formatted at 2026-09-27 05:09:19 on dist-test-slave-hgs0"
server_key: "cef4a6112214afe6aa918039a6a9244a"
server_key_iv: "f6d227dd903fd2a087ad812e651e00d1"
server_key_version: "encryptionkey@0"
I20260927 05:09:25.471808 15568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:25.497815 15568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:25.503059 15568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:25.503396 15568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:25.504838 15568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:25.506454 15568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:25.506564 15568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:25.506739 15568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:25.506822 15568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:25.544211 15568 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32803
I20260927 05:09:25.544294 15695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32803 every 8 connection(s)
I20260927 05:09:25.545816 15568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:09:25.551561 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:25.549260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:42103)
0927 05:09:25.550165 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:25.550422 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":320,"threads_started":1}
W20260927 05:09:25.552336 15696 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46633 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46633: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
I20260927 05:09:25.553328 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:25.552957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:49035)
0927 05:09:25.553119 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:25.553199 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 05:09:25.554287 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:25.553949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:52919)
0927 05:09:25.554088 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:25.554168 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:09:25.554646 15696 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:09:25.555743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 15568
I20260927 05:09:25.556480 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34629
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46633
--webserver_port=35721
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:25.918345 15699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:25.918677 15699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:25.918782 15699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:25.926548 15699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:25.926702 15699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:25.926829 15699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:09:25.937722 15699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35721
--tserver_master_addrs=127.0.97.62:46633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:25.939167 15699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:25.940856 15699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:25.956524 15704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:25.956604 15705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:25.956555 15707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:25.957517 15699 server_base.cc:1034] running on GCE node
I20260927 05:09:25.958216 15699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:25.959308 15699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:25.960546 15699 hybrid_clock.cc:648] HybridClock initialized: now 1790485765960467 us; error 64 us; skew 500 ppm
I20260927 05:09:25.960942 15699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:25 dist-test-slave-hgs0 krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:25.964794 15699 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:09:25.966607 15699 webserver.cc:460] Webserver started at http://127.0.97.3:35721/ using document root <none> and password file <none>
I20260927 05:09:25.967479 15699 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:25.967654 15699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:25.974334 15699 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260927 05:09:25.977638 15714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:25.979300 15699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:09:25.979516 15699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4"
format_stamp: "Formatted at 2026-09-27 05:09:19 on dist-test-slave-hgs0"
server_key: "e8c54c58f7f45df5dc5016f87f7d8da4"
server_key_iv: "220ca133719059cc597f08332988df61"
server_key_version: "encryptionkey@0"
I20260927 05:09:25.980712 15699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:26.002990 15699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:26.008689 15699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:26.009133 15699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:26.010804 15699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:26.012794 15699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:26.012985 15699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:26.013147 15699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:26.013278 15699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:26.058009 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:26.057178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:34257)
0927 05:09:26.057684 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:26.057787 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":227,"threads_started":1}
I20260927 05:09:26.059278 15699 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34629
I20260927 05:09:26.059329 15827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34629 every 8 connection(s)
I20260927 05:09:26.061090 15699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:09:26.062675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 15699
I20260927 05:09:26.067050 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:26.064883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:35063)
0927 05:09:26.065670 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:26.065905 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":327,"threads_started":1}
W20260927 05:09:26.067898 15828 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46633 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46633: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
I20260927 05:09:26.068993 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:26.068616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:44329)
0927 05:09:26.068776 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:26.068865 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 05:09:26.070098 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:26.069728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:50811)
0927 05:09:26.069889 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:26.069972 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 05:09:26.070441 15828 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:09:26.356523 15832 raft_consensus.cc:482] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:09:26.356890 15832 raft_consensus.cc:504] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:26.358769 15832 leader_election.cc:288] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629), c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803)
Sep 27 05:09:26 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 27 05:09:26 dist-test-slave-hgs0 krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:09:26.384922 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:26.359461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60679 (local address 127.0.97.3:34629)
0927 05:09:26.361446 (+ 1985us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:26.361469 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:26.361533 (+ 64us) server_negotiation.cc:407] Connection header received
0927 05:09:26.363651 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:26.363662 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:26.364216 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:26.364539 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:26.367246 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:26.368730 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:26.370844 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:26.371802 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:26.372059 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:26.376811 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:26.376825 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:26.376850 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:26.376923 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:26.380344 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:26.381397 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:26.381408 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:26.381417 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:26.381615 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:26.382124 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:26.382130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:26.382135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:26.382744 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:26.382966 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:26.383994 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0927 05:09:26.384441 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1707,"thread_start_us":1456,"threads_started":1}
I20260927 05:09:26.384984 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:26.359121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34629 (local address 127.0.97.1:60679)
0927 05:09:26.359448 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:26.359493 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:26.360277 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:26.364781 (+ 4504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:26.364795 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:26.365993 (+ 1198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:26.366939 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:26.366966 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:26.369011 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:26.369015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:26.370657 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:26.370674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:26.372046 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:26.372053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:26.372300 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:26.373178 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:26.373214 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:26.376386 (+ 3172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:26.380541 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:26.380553 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:26.380574 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:26.381225 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:26.381763 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:26.381769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:26.381773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:26.381999 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:26.382922 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:26.382931 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:26.383319 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 05:09:26.383631 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 05:09:26.384527 (+ 896us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":456,"spinlock_wait_cycles":47360}
I20260927 05:09:26.384999 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:26.359621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40611 (local address 127.0.97.2:32803)
0927 05:09:26.362438 (+ 2817us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:26.362455 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:26.362514 (+ 59us) server_negotiation.cc:407] Connection header received
0927 05:09:26.364362 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:26.364373 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:26.364922 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:26.365342 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:26.367248 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:26.368746 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:26.370858 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:26.371643 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:26.371900 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:26.377081 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:26.377090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:26.377115 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:26.377161 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:26.380387 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:26.381439 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:26.381446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:26.381452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:26.381634 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:26.382169 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:26.382172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:26.382174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:26.382742 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:26.382966 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:26.383962 (+ 996us) server_negotiation.cc:299] Negotiation successful
0927 05:09:26.384527 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2611,"thread_start_us":2488,"threads_started":1}
I20260927 05:09:26.385792 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:26.359159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32803 (local address 127.0.97.1:40611)
0927 05:09:26.361720 (+ 2561us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:26.361742 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:26.361891 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:26.365339 (+ 3448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:26.365344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:26.366067 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:26.366941 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:26.366968 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:26.368994 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:26.368999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:26.370664 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:26.370680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:26.371889 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:26.371897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:26.372281 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:26.373174 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:26.373216 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:26.376793 (+ 3577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:26.380580 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:26.380584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:26.380587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:26.381237 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:26.381797 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:26.381800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:26.381803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:26.382016 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:26.382921 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:26.382931 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:26.383316 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 05:09:26.383631 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 05:09:26.384359 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2389,"mutex_wait_us":225,"spinlock_wait_cycles":69376,"thread_start_us":2199,"threads_started":1}
W20260927 05:09:26.389436 15452 leader_election.cc:341] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629): Not found: Tablet not found: 38b8ecdffd5f4f23972d448da6d177ff
W20260927 05:09:26.389829 15451 leader_election.cc:341] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803): Not found: Tablet not found: 38b8ecdffd5f4f23972d448da6d177ff
I20260927 05:09:26.390023 15451 leader_election.cc:302] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb4f1418715a4df99a9b830ecf2b963d; no voters: c92009229c9948b7a19d815d1944d5a8, fbc1a0d0116a4303a9cd4e13d44c6ff4
I20260927 05:09:26.390520 15832 raft_consensus.cc:2726] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:09:26.556739 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:26.555370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:52209)
0927 05:09:26.556143 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:26.556356 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":274,"threads_started":1}
I20260927 05:09:27.060603 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:27.059144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:37181)
0927 05:09:27.060306 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:27.060391 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1051,"thread_start_us":945,"threads_started":1}
I20260927 05:09:27.071753 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:27.071025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:38577)
0927 05:09:27.071440 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:27.071557 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":194,"threads_started":1}
I20260927 05:09:27.558670 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:27.557907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:59117)
0927 05:09:27.558412 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:27.558503 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":267,"threads_started":1}
I20260927 05:09:28.062299 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:28.061511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:39199)
0927 05:09:28.062035 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:28.062110 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
I20260927 05:09:28.073396 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:28.072642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:60357)
0927 05:09:28.073126 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:28.073201 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":292,"threads_started":1}
I20260927 05:09:28.259399 15842 raft_consensus.cc:482] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:09:28.259629 15842 raft_consensus.cc:504] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:28.260426 15842 leader_election.cc:288] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629), c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803)
W20260927 05:09:28.261453 15452 leader_election.cc:341] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629): Not found: Tablet not found: 38b8ecdffd5f4f23972d448da6d177ff
W20260927 05:09:28.261620 15451 leader_election.cc:341] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803): Not found: Tablet not found: 38b8ecdffd5f4f23972d448da6d177ff
I20260927 05:09:28.261757 15451 leader_election.cc:302] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb4f1418715a4df99a9b830ecf2b963d; no voters: c92009229c9948b7a19d815d1944d5a8, fbc1a0d0116a4303a9cd4e13d44c6ff4
I20260927 05:09:28.262053 15842 raft_consensus.cc:2726] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:09:28.560933 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:28.559999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:48941)
0927 05:09:28.560617 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:28.560723 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":318,"threads_started":1}
I20260927 05:09:29.064266 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:29.063421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:44449)
0927 05:09:29.063992 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:29.064087 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":302,"threads_started":1}
I20260927 05:09:29.075367 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:29.074560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:51215)
0927 05:09:29.075078 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:29.075176 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":256,"threads_started":1}
I20260927 05:09:29.562744 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:29.561982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:52865)
0927 05:09:29.562466 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:29.562553 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":236,"threads_started":1}
I20260927 05:09:30.066200 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:30.065351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:37641)
0927 05:09:30.065902 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:30.065989 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":228,"threads_started":1}
I20260927 05:09:30.077522 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:30.076574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:46087)
0927 05:09:30.077207 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:30.077321 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":339,"threads_started":1}
I20260927 05:09:30.565037 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:30.564065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:46211)
0927 05:09:30.564670 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:30.564757 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":335,"threads_started":1}
I20260927 05:09:30.737417 15850 raft_consensus.cc:482] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:09:30.737661 15850 raft_consensus.cc:504] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:30.738380 15850 leader_election.cc:288] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629), c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803)
W20260927 05:09:30.739627 15451 leader_election.cc:341] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803): Not found: Tablet not found: 38b8ecdffd5f4f23972d448da6d177ff
W20260927 05:09:30.739909 15452 leader_election.cc:341] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629): Not found: Tablet not found: 38b8ecdffd5f4f23972d448da6d177ff
I20260927 05:09:30.740114 15452 leader_election.cc:302] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb4f1418715a4df99a9b830ecf2b963d; no voters: c92009229c9948b7a19d815d1944d5a8, fbc1a0d0116a4303a9cd4e13d44c6ff4
I20260927 05:09:30.740456 15850 raft_consensus.cc:2726] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:09:31.068171 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:31.067217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:45339)
0927 05:09:31.067803 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:31.067913 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":267,"threads_started":1}
I20260927 05:09:31.079541 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:31.078659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:36085)
0927 05:09:31.079197 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:31.079337 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":296,"threads_started":1}
I20260927 05:09:31.567190 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:31.566124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:53949)
0927 05:09:31.566756 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:31.566867 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":325,"threads_started":1}
I20260927 05:09:32.070408 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:32.069262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:52451)
0927 05:09:32.070061 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:32.070171 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":288,"threads_started":1}
I20260927 05:09:32.081602 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:32.080672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:34071)
0927 05:09:32.081317 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:32.081409 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":393,"threads_started":1}
I20260927 05:09:32.569228 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:32.568409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:55321)
0927 05:09:32.568944 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:32.569031 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":226,"threads_started":1}
I20260927 05:09:33.072507 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:33.071548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:47437)
0927 05:09:33.072193 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:33.072322 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":278,"threads_started":1}
I20260927 05:09:33.083557 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:33.082651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:58261)
0927 05:09:33.083250 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:33.083351 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":315,"threads_started":1}
I20260927 05:09:33.571008 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:33.570253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:45921)
0927 05:09:33.570708 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:33.570807 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":227,"threads_started":1}
I20260927 05:09:34.074569 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:34.073662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:59419)
0927 05:09:34.074244 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:34.074378 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":345,"threads_started":1}
I20260927 05:09:34.085860 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:34.084738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:49883)
0927 05:09:34.085371 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:34.085494 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":309,"threads_started":1}
I20260927 05:09:34.573086 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:34.572317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:58047)
0927 05:09:34.572821 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:34.572915 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":222,"threads_started":1}
I20260927 05:09:35.076767 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:35.075810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:53989)
0927 05:09:35.076433 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:35.076523 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":370,"threads_started":1}
I20260927 05:09:35.087759 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:35.086957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:37699)
0927 05:09:35.087479 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:35.087574 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":264,"threads_started":1}
I20260927 05:09:35.103030 15865 raft_consensus.cc:482] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:09:35.103268 15865 raft_consensus.cc:504] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc1a0d0116a4303a9cd4e13d44c6ff4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34629 } } peers { permanent_uuid: "fb4f1418715a4df99a9b830ecf2b963d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41047 } } peers { permanent_uuid: "c92009229c9948b7a19d815d1944d5a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32803 } }
I20260927 05:09:35.104115 15865 leader_election.cc:288] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629), c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803)
W20260927 05:09:35.105149 15452 leader_election.cc:341] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbc1a0d0116a4303a9cd4e13d44c6ff4 (127.0.97.3:34629): Not found: Tablet not found: 38b8ecdffd5f4f23972d448da6d177ff
W20260927 05:09:35.105366 15451 leader_election.cc:341] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c92009229c9948b7a19d815d1944d5a8 (127.0.97.2:32803): Not found: Tablet not found: 38b8ecdffd5f4f23972d448da6d177ff
I20260927 05:09:35.105511 15451 leader_election.cc:302] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb4f1418715a4df99a9b830ecf2b963d; no voters: c92009229c9948b7a19d815d1944d5a8, fbc1a0d0116a4303a9cd4e13d44c6ff4
I20260927 05:09:35.105741 15865 raft_consensus.cc:2726] T 38b8ecdffd5f4f23972d448da6d177ff P fb4f1418715a4df99a9b830ecf2b963d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:09:35.574808 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:35.574100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:54437)
0927 05:09:35.574539 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:35.574624 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":239,"threads_started":1}
I20260927 05:09:36.078404 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:36.077691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:46161)
0927 05:09:36.078129 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:36.078214 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":233,"threads_started":1}
I20260927 05:09:36.089432 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:36.088674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:45347)
0927 05:09:36.089173 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:36.089249 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":298,"threads_started":1}
I20260927 05:09:36.576815 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:36.575885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:57471)
0927 05:09:36.576544 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:36.576637 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":352,"threads_started":1}
I20260927 05:09:37.080266 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:37.079419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:40453)
0927 05:09:37.079915 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:37.080088 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":248,"threads_started":1}
I20260927 05:09:37.091112 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:37.090311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:48797)
0927 05:09:37.090866 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:37.090955 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":300,"threads_started":1}
I20260927 05:09:37.578526 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:37.577819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:54875)
0927 05:09:37.578258 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:37.578346 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260927 05:09:38.082105 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:38.081233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:44873)
0927 05:09:38.081798 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:38.081901 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":278,"threads_started":1}
I20260927 05:09:38.093040 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:38.092218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:40365)
0927 05:09:38.092753 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:38.092849 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":300,"threads_started":1}
I20260927 05:09:38.580348 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:38.579502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:53327)
0927 05:09:38.580067 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:38.580158 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":262,"threads_started":1}
I20260927 05:09:39.084180 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:39.083213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:48621)
0927 05:09:39.083751 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:39.083925 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":255,"threads_started":1}
I20260927 05:09:39.095093 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:39.094192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:56219)
0927 05:09:39.094800 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:39.094915 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":269,"threads_started":1}
I20260927 05:09:39.582166 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:39.581354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:56925)
0927 05:09:39.581861 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:39.581955 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":269,"threads_started":1}
I20260927 05:09:40.086295 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:40.085446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:58665)
0927 05:09:40.086029 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:40.086111 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":319,"threads_started":1}
I20260927 05:09:40.097093 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:40.096312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:36595)
0927 05:09:40.096788 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:40.096877 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":231,"threads_started":1}
I20260927 05:09:40.584314 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:40.583506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.2:53519)
0927 05:09:40.584031 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:40.584136 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":222,"threads_started":1}
I20260927 05:09:41.069947 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 15400
I20260927 05:09:41.088382 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:41.087286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.1:38291)
0927 05:09:41.087922 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:41.088086 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":333,"threads_started":1}
I20260927 05:09:41.088912 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:41.071468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52528 (local address 127.0.97.1:41047)
0927 05:09:41.072238 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:41.072247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:41.072283 (+ 36us) server_negotiation.cc:407] Connection header received
0927 05:09:41.072350 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:41.072357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:41.072665 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:41.072871 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:41.074730 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:41.076039 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:41.077534 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:41.078249 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:41.078427 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:41.081616 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:41.081632 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:41.081650 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:41.081696 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:41.085466 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:41.086314 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:41.086325 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:41.086328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:41.086492 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:41.086961 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:41.086966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:41.086970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:41.087405 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:41.087806 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:41.088157 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 05:09:41.088484 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":391,"threads_started":1}
I20260927 05:09:41.090271 15560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:41.099464 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:41.098444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46633 (local address 127.0.97.3:34165)
0927 05:09:41.099071 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:41.099206 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":297,"threads_started":1}
I20260927 05:09:41.185403 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 15400
I20260927 05:09:41.203003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 15568
I20260927 05:09:41.218119 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:41.203914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.0.97.2:32803)
0927 05:09:41.204491 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:41.204495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:41.204514 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:09:41.204575 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:41.204579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:41.204640 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:41.204739 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:41.206463 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:41.207356 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:41.208809 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:41.209293 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:41.209404 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:41.212370 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:41.212376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:41.212379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:41.212421 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:41.215488 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:41.216337 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:41.216341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:41.216344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:41.216487 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:41.216832 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:41.216835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:41.216837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:41.217108 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:41.217334 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:41.217600 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 05:09:41.217810 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":319,"threads_started":1}
I20260927 05:09:41.218878 15691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:41.306037 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 15568
I20260927 05:09:41.324129 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 15699
I20260927 05:09:41.338797 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:41.325073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.0.97.3:34629)
0927 05:09:41.325602 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:41.325607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:41.325625 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:09:41.325680 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:41.325683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:41.325786 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:41.325878 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:41.327197 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:41.328142 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:41.329506 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:41.329952 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:41.330097 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:41.332926 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:41.332931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:41.332934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:41.332974 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:41.335869 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:41.336838 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:41.336843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:41.336846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:41.337010 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:41.337374 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:41.337377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:41.337379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:41.337664 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:41.337945 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:41.338263 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 05:09:41.338458 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":284,"threads_started":1}
I20260927 05:09:41.339602 15822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:41.430128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 15699
2026-09-27T05:09:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24921 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:09:41 dist-test-slave-hgs0 krb5kdc[15892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:09:41 dist-test-slave-hgs0 krb5kdc[15892](info): set up 2 sockets
Sep 27 05:09:41 dist-test-slave-hgs0 krb5kdc[15892](info): commencing operation
krb5kdc: starting...
W20260927 05:09:42.560622 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:09:42 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:09:42.824896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34651
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:43.198444 15905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:43.198874 15905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:43.199059 15905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:43.211388 15905 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 05:09:43.211558 15905 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 05:09:43.211697 15905 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 05:09:43.211803 15905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:09:43.211977 15905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:43.212138 15905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:43.212255 15905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:09:43.212384 15905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:09:43.224314 15905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:43.225863 15905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:43.227454 15905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:43.241142 15910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:43.241437 15911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:43.241935 15913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:43.242753 15905 server_base.cc:1034] running on GCE node
I20260927 05:09:43.243713 15905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:43.245466 15905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:43.246753 15905 hybrid_clock.cc:648] HybridClock initialized: now 1790485783246655 us; error 76 us; skew 500 ppm
I20260927 05:09:43.247165 15905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:43 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:43.250808 15905 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:09:43.252694 15905 webserver.cc:460] Webserver started at http://127.0.97.62:36865/ using document root <none> and password file <none>
I20260927 05:09:43.253466 15905 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:43.253561 15905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:43.253849 15905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:43.257206 15905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "5587b12ee129401e83435f9cd8c84429"
format_stamp: "Formatted at 2026-09-27 05:09:43 on dist-test-slave-hgs0"
server_key: "6f911aebbc646d1dd45dd6e967cb51dc"
server_key_iv: "3e315f95f7d72a853bc237cb19e9ac6d"
server_key_version: "encryptionkey@0"
I20260927 05:09:43.258046 15905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "5587b12ee129401e83435f9cd8c84429"
format_stamp: "Formatted at 2026-09-27 05:09:43 on dist-test-slave-hgs0"
server_key: "6f911aebbc646d1dd45dd6e967cb51dc"
server_key_iv: "3e315f95f7d72a853bc237cb19e9ac6d"
server_key_version: "encryptionkey@0"
I20260927 05:09:43.264099 15905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 05:09:43.268255 15920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:43.270020 15905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:09:43.270255 15905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "5587b12ee129401e83435f9cd8c84429"
format_stamp: "Formatted at 2026-09-27 05:09:43 on dist-test-slave-hgs0"
server_key: "6f911aebbc646d1dd45dd6e967cb51dc"
server_key_iv: "3e315f95f7d72a853bc237cb19e9ac6d"
server_key_version: "encryptionkey@0"
I20260927 05:09:43.270546 15905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:43.294806 15905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:43.299046 15905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:43.299351 15905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:43.319344 15905 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34651
I20260927 05:09:43.319382 15971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34651 every 8 connection(s)
I20260927 05:09:43.320886 15905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:09:43.321949 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 15905
I20260927 05:09:43.322194 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:09:43.323033 388 external_mini_cluster.cc:1428] Setting key 45bb30c1964e4737fe77fcc34de17bf6
I20260927 05:09:43.326282 15972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 05:09:43 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:43.337858 15972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Bootstrap starting.
I20260927 05:09:43.341717 15972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:43.342854 15972 log.cc:824] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:43.346069 15972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: No bootstrap required, opened a new log
I20260927 05:09:43.346625 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:43.325267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.0.97.62:34651)
0927 05:09:43.326049 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:43.326063 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:43.326103 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:09:43.327832 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:43.327855 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:43.328350 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:43.328801 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:43.330381 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:43.331762 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:43.333278 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:43.333990 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:43.334214 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:43.338777 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:43.338790 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:43.338816 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:43.338862 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:43.342160 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:43.342951 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:43.342959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:43.342966 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:43.343179 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:43.343588 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:43.343593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:43.343596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:43.344271 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:43.344641 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:43.345106 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 05:09:43.345553 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":347,"threads_started":1}
I20260927 05:09:43.354107 15972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:09:43.354552 15972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:43.354669 15972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5587b12ee129401e83435f9cd8c84429, State: Initialized, Role: FOLLOWER
I20260927 05:09:43.355434 15972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:09:43.355728 15972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:09:43.355871 15972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:09:43.356122 15972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:43.358489 15972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:09:43.358997 15972 leader_election.cc:302] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5587b12ee129401e83435f9cd8c84429; no voters:
I20260927 05:09:43.359615 15972 leader_election.cc:288] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:09:43.359746 15977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:09:43.361274 15977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 1 LEADER]: Becoming Leader. State: Replica: 5587b12ee129401e83435f9cd8c84429, State: Running, Role: LEADER
I20260927 05:09:43.361881 15977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:09:43.362165 15972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:09:43.364997 15979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5587b12ee129401e83435f9cd8c84429. Latest consensus state: current_term: 1 leader_uuid: "5587b12ee129401e83435f9cd8c84429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } } }
I20260927 05:09:43.365653 15979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:43.365093 15978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5587b12ee129401e83435f9cd8c84429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } } }
I20260927 05:09:43.366163 15978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: This master's current role is: LEADER
I20260927 05:09:43.368007 15986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:09:43.373607 15986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:09:43.383447 15986 catalog_manager.cc:1347] Generated new cluster ID: 123b2d849b7c4a0eb9741b94d6fc19dc
I20260927 05:09:43.383599 15986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:09:43.413761 15986 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:09:43.415818 15986 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:09:43.428130 15986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Generated new TSK 0
I20260927 05:09:43.429373 15986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:09:43.497516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34651
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:43.862866 15998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:43.863222 15998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:43.863415 15998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:43.871268 15998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:43.871423 15998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:43.871541 15998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:09:43.883188 15998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:43.884738 15998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:43.886134 15998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:43.900561 16006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:43.900586 16004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:43.900612 16003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:43.901574 15998 server_base.cc:1034] running on GCE node
I20260927 05:09:43.902379 15998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:43.903462 15998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:43.904727 15998 hybrid_clock.cc:648] HybridClock initialized: now 1790485783904671 us; error 58 us; skew 500 ppm
I20260927 05:09:43.905141 15998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:43 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:43.909192 15998 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:09:43.911209 15998 webserver.cc:460] Webserver started at http://127.0.97.1:45937/ using document root <none> and password file <none>
I20260927 05:09:43.912096 15998 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:43.912284 15998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:43.912628 15998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:43.916038 15998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "00d4ba2781e54dc2955557054383c27d"
format_stamp: "Formatted at 2026-09-27 05:09:43 on dist-test-slave-hgs0"
server_key: "d39a9ff92fe0505df159b031081810e2"
server_key_iv: "e68cd55d198bba07f0856f2caa3eb951"
server_key_version: "encryptionkey@0"
I20260927 05:09:43.916932 15998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "00d4ba2781e54dc2955557054383c27d"
format_stamp: "Formatted at 2026-09-27 05:09:43 on dist-test-slave-hgs0"
server_key: "d39a9ff92fe0505df159b031081810e2"
server_key_iv: "e68cd55d198bba07f0856f2caa3eb951"
server_key_version: "encryptionkey@0"
I20260927 05:09:43.923002 15998 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:09:43.927309 16013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:43.929299 15998 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:09:43.929548 15998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "00d4ba2781e54dc2955557054383c27d"
format_stamp: "Formatted at 2026-09-27 05:09:43 on dist-test-slave-hgs0"
server_key: "d39a9ff92fe0505df159b031081810e2"
server_key_iv: "e68cd55d198bba07f0856f2caa3eb951"
server_key_version: "encryptionkey@0"
I20260927 05:09:43.929824 15998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:43.984793 15998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:43.989509 15998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:43.989892 15998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:43.991103 15998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:43.992812 15998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:43.992942 15998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:43.993111 15998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:43.993193 15998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:44.028534 15998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33473
I20260927 05:09:44.028594 16125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33473 every 8 connection(s)
I20260927 05:09:44.029886 15998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:09:44.037040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 15998
I20260927 05:09:44.037257 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:09:44.037899 388 external_mini_cluster.cc:1428] Setting key f9b0b5d305ca7a77db739a1b22323ac8
Sep 27 05:09:44 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:44.052683 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:44.032605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49377 (local address 127.0.97.62:34651)
0927 05:09:44.033054 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:44.033059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:44.034158 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 05:09:44.036554 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:44.036560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:44.036664 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:44.036793 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:44.038721 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:44.039606 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:44.041900 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:44.042312 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:44.042470 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:44.046280 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:44.046286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:44.046290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:44.046328 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:44.049253 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:44.050127 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:44.050132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:44.050134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:44.050283 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:44.050753 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:44.050758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:44.050760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:44.051054 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:44.051242 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:44.052052 (+ 810us) server_negotiation.cc:299] Negotiation successful
0927 05:09:44.052307 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":241,"threads_started":1}
I20260927 05:09:44.053867 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:44.033214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:49377)
0927 05:09:44.033988 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:44.034038 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:44.036218 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:44.037055 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:44.037065 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:44.037745 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:44.038470 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:44.038505 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:44.039775 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:44.039781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:44.041739 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:44.041752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:44.042570 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:44.042578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:44.042847 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:44.043590 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:44.043615 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:44.045999 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:44.049442 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:44.049452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:44.049474 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:44.049965 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:44.050400 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:44.050405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:44.050409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:44.050614 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:44.051198 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:44.051209 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:44.051620 (+ 411us) client_negotiation.cc:769] Sending connection context
0927 05:09:44.051907 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 05:09:44.052551 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":271,"threads_started":1}
I20260927 05:09:44.055892 16126 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34651
I20260927 05:09:44.056347 16126 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:44.057210 16126 heartbeater.cc:511] Master 127.0.97.62:34651 requested a full tablet report, sending...
I20260927 05:09:44.060278 15937 ts_manager.cc:194] Registered new tserver with Master: 00d4ba2781e54dc2955557054383c27d (127.0.97.1:33473)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:09:44.062867 15937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:09:44.096167 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34651
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:44.441967 16132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:44.442288 16132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:44.442471 16132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:44.449887 16132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:44.450054 16132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:44.450227 16132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:09:44.460310 16132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:44.461761 16132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:44.463258 16132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:44.476958 16137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:44.476888 16140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:44.476923 16138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:44.477702 16132 server_base.cc:1034] running on GCE node
I20260927 05:09:44.478339 16132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:44.479367 16132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:44.480582 16132 hybrid_clock.cc:648] HybridClock initialized: now 1790485784480530 us; error 42 us; skew 500 ppm
I20260927 05:09:44.480871 16132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:44 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:44.484038 16132 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:09:44.485612 16132 webserver.cc:460] Webserver started at http://127.0.97.2:46093/ using document root <none> and password file <none>
I20260927 05:09:44.486317 16132 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:44.486464 16132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:44.486780 16132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:44.489770 16132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "68ef1e853a264b39a83fc939de8ef703"
format_stamp: "Formatted at 2026-09-27 05:09:44 on dist-test-slave-hgs0"
server_key: "207b750ad9af11bb902737205bed0edb"
server_key_iv: "e28d54eac57cb590c0bec82aceb7c4e2"
server_key_version: "encryptionkey@0"
I20260927 05:09:44.490553 16132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "68ef1e853a264b39a83fc939de8ef703"
format_stamp: "Formatted at 2026-09-27 05:09:44 on dist-test-slave-hgs0"
server_key: "207b750ad9af11bb902737205bed0edb"
server_key_iv: "e28d54eac57cb590c0bec82aceb7c4e2"
server_key_version: "encryptionkey@0"
I20260927 05:09:44.496218 16132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 05:09:44.500214 16147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:44.501696 16132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:09:44.501937 16132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "68ef1e853a264b39a83fc939de8ef703"
format_stamp: "Formatted at 2026-09-27 05:09:44 on dist-test-slave-hgs0"
server_key: "207b750ad9af11bb902737205bed0edb"
server_key_iv: "e28d54eac57cb590c0bec82aceb7c4e2"
server_key_version: "encryptionkey@0"
I20260927 05:09:44.502161 16132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:44.522261 16132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:44.526525 16132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:44.526823 16132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:44.527835 16132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:44.529206 16132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:44.529284 16132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:44.529452 16132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:44.529505 16132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:44.563515 16132 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44465
I20260927 05:09:44.563561 16259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44465 every 8 connection(s)
I20260927 05:09:44.564947 16132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:09:44.571122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 16132
I20260927 05:09:44.571346 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:09:44.571913 388 external_mini_cluster.cc:1428] Setting key 0a515f20f3853b91ba0d1d0a71c724f1
Sep 27 05:09:44 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:44.586681 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:44.567631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56863 (local address 127.0.97.62:34651)
0927 05:09:44.568132 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:44.568137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:44.569213 (+ 1076us) server_negotiation.cc:407] Connection header received
0927 05:09:44.571137 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:44.571145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:44.571254 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:44.571410 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:44.573185 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:44.574155 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:44.575979 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:44.576777 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:44.576927 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:44.580477 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:44.580486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:44.580494 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:44.580543 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:44.583370 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:44.584351 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:44.584355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:44.584357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:44.584479 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:44.584958 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:44.584961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:44.584963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:44.585211 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:44.585338 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:44.586216 (+ 878us) server_negotiation.cc:299] Negotiation successful
0927 05:09:44.586399 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":226,"threads_started":1}
I20260927 05:09:44.588261 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:44.568316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:56863)
0927 05:09:44.569035 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:44.569085 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:44.570823 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:44.571685 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:44.571695 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:44.572359 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:44.572978 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:44.573007 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:44.574334 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:44.574338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:44.575809 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:44.575821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:44.577004 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:44.577009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:44.577218 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:44.577938 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:44.577964 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:44.580213 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:44.583518 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:44.583531 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:44.583552 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:44.584215 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:44.584586 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:44.584593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:44.584599 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:44.584847 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:44.585342 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:44.585353 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:44.585828 (+ 475us) client_negotiation.cc:769] Sending connection context
0927 05:09:44.586147 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 05:09:44.586744 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":275,"threads_started":1}
I20260927 05:09:44.589809 16260 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34651
I20260927 05:09:44.590161 16260 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:44.590960 16260 heartbeater.cc:511] Master 127.0.97.62:34651 requested a full tablet report, sending...
I20260927 05:09:44.592661 15937 ts_manager.cc:194] Registered new tserver with Master: 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
I20260927 05:09:44.593994 15937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56863
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:09:44.628131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34651
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:09:44.965481 16266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:09:44.965790 16266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:09:44.966004 16266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:09:44.973317 16266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:09:44.973429 16266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:09:44.973627 16266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:09:44.984297 16266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:09:44.985719 16266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:09:44.987107 16266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:09:45.000448 16271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:45.001036 16272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:09:45.001402 16274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:09:45.002075 16266 server_base.cc:1034] running on GCE node
I20260927 05:09:45.002624 16266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:09:45.003463 16266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:09:45.004648 16266 hybrid_clock.cc:648] HybridClock initialized: now 1790485785004596 us; error 66 us; skew 500 ppm
I20260927 05:09:45.004956 16266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:09:45 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:09:45.014775 16266 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:09:45.016535 16266 webserver.cc:460] Webserver started at http://127.0.97.3:43289/ using document root <none> and password file <none>
I20260927 05:09:45.017251 16266 fs_manager.cc:359] Metadata directory not provided
I20260927 05:09:45.017407 16266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:09:45.017727 16266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:09:45.020713 16266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "9a928f8d66224576be6e2d8caf31ce04"
format_stamp: "Formatted at 2026-09-27 05:09:45 on dist-test-slave-hgs0"
server_key: "d6649204c886593805eb58290581e2e3"
server_key_iv: "33c608f44d25cc47975a38a590c00c8d"
server_key_version: "encryptionkey@0"
I20260927 05:09:45.021486 16266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9a928f8d66224576be6e2d8caf31ce04"
format_stamp: "Formatted at 2026-09-27 05:09:45 on dist-test-slave-hgs0"
server_key: "d6649204c886593805eb58290581e2e3"
server_key_iv: "33c608f44d25cc47975a38a590c00c8d"
server_key_version: "encryptionkey@0"
I20260927 05:09:45.026678 16266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 05:09:45.030670 16281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:45.032184 16266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:09:45.032361 16266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "9a928f8d66224576be6e2d8caf31ce04"
format_stamp: "Formatted at 2026-09-27 05:09:45 on dist-test-slave-hgs0"
server_key: "d6649204c886593805eb58290581e2e3"
server_key_iv: "33c608f44d25cc47975a38a590c00c8d"
server_key_version: "encryptionkey@0"
I20260927 05:09:45.032531 16266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:09:45.055794 16266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:09:45.060491 16266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:09:45.060788 16266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:09:45.061868 16266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:09:45.063381 16266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:09:45.063516 16266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:45.063642 16266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:09:45.063702 16266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:09:45.066505 16126 heartbeater.cc:503] Master 127.0.97.62:34651 was elected leader, sending a full tablet report...
I20260927 05:09:45.096588 16266 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38555
I20260927 05:09:45.096606 16393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38555 every 8 connection(s)
I20260927 05:09:45.097990 16266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:09:45.102828 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 16266
I20260927 05:09:45.103024 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:09:45.103484 388 external_mini_cluster.cc:1428] Setting key fc4eb82ee2ac73122fc172032fabc8c9
Sep 27 05:09:45 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:09:45.118345 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.100578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53307 (local address 127.0.97.62:34651)
0927 05:09:45.101020 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.101024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.101904 (+ 880us) server_negotiation.cc:407] Connection header received
0927 05:09:45.103966 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.103972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.104083 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.104204 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:45.106002 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.107031 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.108677 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.109088 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.109185 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.112680 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:45.112684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:45.112688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:45.112723 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:45.115210 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:45.115917 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:45.115921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:45.115924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:45.116100 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:45.116492 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:45.116495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:45.116497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:45.116762 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:45.116881 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.117698 (+ 817us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.117936 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":234,"threads_started":1}
I20260927 05:09:45.119383 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.101043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:53307)
0927 05:09:45.101732 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.101770 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:45.103653 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:45.104495 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:45.104506 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:45.105090 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:09:45.105797 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.105847 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.107169 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.107173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.108543 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.108556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.109282 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.109286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.109489 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.110141 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:09:45.110165 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:09:45.112446 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:09:45.115344 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:45.115351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:45.115367 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:45.115793 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:45.116197 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:09:45.116200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:09:45.116202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:09:45.116393 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:09:45.116901 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:09:45.116909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:09:45.117310 (+ 401us) client_negotiation.cc:769] Sending connection context
0927 05:09:45.117594 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 05:09:45.118166 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":265,"threads_started":1}
I20260927 05:09:45.120859 16394 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34651
I20260927 05:09:45.121228 16394 heartbeater.cc:464] Registering TS with master...
I20260927 05:09:45.122028 16394 heartbeater.cc:511] Master 127.0.97.62:34651 requested a full tablet report, sending...
I20260927 05:09:45.123574 15937 ts_manager.cc:194] Registered new tserver with Master: 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555)
I20260927 05:09:45.124477 15937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53307
I20260927 05:09:45.131521 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:09:45.149816 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.136429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.97.62:34651)
0927 05:09:45.136657 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.136661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.137061 (+ 400us) server_negotiation.cc:407] Connection header received
0927 05:09:45.137282 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.137288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.137375 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.137481 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:45.138906 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.139630 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.141124 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.141562 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.141701 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.144261 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:45.144267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:45.144270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:45.144307 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:45.146889 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:45.147671 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:45.147677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:45.147681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:45.147819 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:45.148239 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:45.148245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:45.148248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:45.148619 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:45.148886 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.149220 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.149481 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 05:09:45.154453 15937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:09:45.162856 15937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:09:45.196794 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.176257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.0.97.3:38555)
0927 05:09:45.188613 (+ 12356us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.188622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.188657 (+ 35us) server_negotiation.cc:407] Connection header received
0927 05:09:45.188761 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.188769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.188995 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.189546 (+ 551us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:45.189971 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.191501 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.193906 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.195457 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.195585 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.195816 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.196256 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.196442 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12172,"thread_start_us":12028,"threads_started":1}
I20260927 05:09:45.197326 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.180076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:34330)
0927 05:09:45.187826 (+ 7750us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.187841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:45.188006 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:45.188707 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:45.188712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:45.188762 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:45.189247 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.189260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.191412 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.191418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.193942 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.193956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.195685 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.195694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.195957 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.196052 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 05:09:45.196196 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 05:09:45.196468 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7623,"mutex_wait_us":206,"spinlock_wait_cycles":5888,"thread_start_us":5874,"threads_started":1}
I20260927 05:09:45.197420 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.175824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38555 (local address 127.0.0.1:53148)
0927 05:09:45.176948 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.177005 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:45.177234 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:45.189230 (+ 11996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:45.189235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:45.189259 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:45.189760 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.189776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.191642 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.191649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.193732 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.193748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.195706 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.195711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.195923 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.196052 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 05:09:45.196196 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 05:09:45.196467 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":108,"thread_start_us":11735,"threads_started":1}
I20260927 05:09:45.198123 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.180832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.0.97.1:33473)
0927 05:09:45.187091 (+ 6259us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.187103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.187971 (+ 868us) server_negotiation.cc:407] Connection header received
0927 05:09:45.188193 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.188204 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.188474 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.189360 (+ 886us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:45.189609 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.191273 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.194103 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.195455 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.195585 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.195828 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.197073 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.197240 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6073,"thread_start_us":5897,"threads_started":1}
I20260927 05:09:45.199631 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.176767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.0.97.2:44465)
0927 05:09:45.187503 (+ 10736us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.187516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.187555 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:09:45.187695 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.187708 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.188082 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.189069 (+ 987us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:45.189921 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.190940 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.195917 (+ 4977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.197442 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.197585 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.197790 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.199122 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.199307 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10520,"thread_start_us":10364,"threads_started":1}
I20260927 05:09:45.200227 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.176102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44465 (local address 127.0.0.1:57518)
0927 05:09:45.186334 (+ 10232us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.186361 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:45.186522 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:45.188590 (+ 2068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:45.188597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:45.188629 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:45.189706 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.189725 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.191073 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.191080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.193942 (+ 2862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.193956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.197662 (+ 3706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.197669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.197845 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.197885 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 05:09:45.197978 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 05:09:45.198108 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10041,"mutex_wait_us":518,"thread_start_us":882,"threads_started":1}
I20260927 05:09:45.201522 16329 tablet_service.cc:1467] Processing CreateTablet for tablet 256e623ed7aa4640a612e6b005431d8e (DEFAULT_TABLE table=test-table [id=cf1370982e4641eeb15bbffd5de8e21c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:45.201527 16061 tablet_service.cc:1467] Processing CreateTablet for tablet 256e623ed7aa4640a612e6b005431d8e (DEFAULT_TABLE table=test-table [id=cf1370982e4641eeb15bbffd5de8e21c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:45.202714 16329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256e623ed7aa4640a612e6b005431d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:45.202725 16061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256e623ed7aa4640a612e6b005431d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:45.206183 16195 tablet_service.cc:1467] Processing CreateTablet for tablet 256e623ed7aa4640a612e6b005431d8e (DEFAULT_TABLE table=test-table [id=cf1370982e4641eeb15bbffd5de8e21c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:09:45.207556 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256e623ed7aa4640a612e6b005431d8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:09:45.214249 16413 tablet_bootstrap.cc:492] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Bootstrap starting.
I20260927 05:09:45.217561 16413 tablet_bootstrap.cc:654] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:45.218542 16413 log.cc:824] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:45.220508 16414 tablet_bootstrap.cc:492] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: Bootstrap starting.
I20260927 05:09:45.221658 16413 tablet_bootstrap.cc:492] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: No bootstrap required, opened a new log
I20260927 05:09:45.222062 16413 ts_tablet_manager.cc:1397] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 05:09:45.222296 16416 tablet_bootstrap.cc:492] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: Bootstrap starting.
I20260927 05:09:45.224086 16414 tablet_bootstrap.cc:654] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:45.225080 16416 tablet_bootstrap.cc:654] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: Neither blocks nor log segments found. Creating new log.
I20260927 05:09:45.225302 16414 log.cc:824] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:45.226243 16416 log.cc:824] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: Log is configured to *not* fsync() on all Append() calls
I20260927 05:09:45.228345 16414 tablet_bootstrap.cc:492] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: No bootstrap required, opened a new log
I20260927 05:09:45.228607 16414 ts_tablet_manager.cc:1397] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 05:09:45.228722 16416 tablet_bootstrap.cc:492] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: No bootstrap required, opened a new log
I20260927 05:09:45.228979 16416 ts_tablet_manager.cc:1397] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 05:09:45.232685 16413 raft_consensus.cc:348] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } }
I20260927 05:09:45.233160 16413 raft_consensus.cc:374] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:45.233251 16413 raft_consensus.cc:729] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d4ba2781e54dc2955557054383c27d, State: Initialized, Role: FOLLOWER
I20260927 05:09:45.233929 16413 consensus_queue.cc:260] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } }
I20260927 05:09:45.237195 16413 ts_tablet_manager.cc:1428] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260927 05:09:45.238454 16414 raft_consensus.cc:348] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } }
I20260927 05:09:45.238859 16414 raft_consensus.cc:374] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:45.239108 16414 raft_consensus.cc:729] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a928f8d66224576be6e2d8caf31ce04, State: Initialized, Role: FOLLOWER
I20260927 05:09:45.239785 16414 consensus_queue.cc:260] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } }
I20260927 05:09:45.240163 16416 raft_consensus.cc:348] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } }
I20260927 05:09:45.240651 16416 raft_consensus.cc:374] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:09:45.240957 16394 heartbeater.cc:503] Master 127.0.97.62:34651 was elected leader, sending a full tablet report...
I20260927 05:09:45.241571 16414 ts_tablet_manager.cc:1428] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 05:09:45.241858 16416 raft_consensus.cc:729] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ef1e853a264b39a83fc939de8ef703, State: Initialized, Role: FOLLOWER
I20260927 05:09:45.242481 16416 consensus_queue.cc:260] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } }
I20260927 05:09:45.243582 16260 heartbeater.cc:503] Master 127.0.97.62:34651 was elected leader, sending a full tablet report...
I20260927 05:09:45.244287 16416 ts_tablet_manager.cc:1428] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260927 05:09:45.282456 16127 tablet.cc:2369] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:09:45.317107 16261 tablet.cc:2369] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:09:45.345536 16419 raft_consensus.cc:482] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:09:45.345798 16419 raft_consensus.cc:504] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } }
I20260927 05:09:45.347363 16419 leader_election.cc:288] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555), 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
W20260927 05:09:45.349692 16395 tablet.cc:2369] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:09:45.356949 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.348261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36089 (local address 127.0.97.2:44465)
0927 05:09:45.348627 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.348634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.348877 (+ 243us) server_negotiation.cc:407] Connection header received
0927 05:09:45.349118 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.349123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.349203 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.349300 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:45.350059 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.351201 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.354463 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.355689 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.355834 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.355925 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.356548 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.356681 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260927 05:09:45.356938 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.348335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57397 (local address 127.0.97.3:38555)
0927 05:09:45.348805 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.348812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.348837 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:09:45.349038 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.349042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.349107 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.349258 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:09:45.350170 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.351254 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.354470 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.355692 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.355825 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.355928 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.356548 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.356681 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322}
I20260927 05:09:45.357173 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.347858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44465 (local address 127.0.97.1:36089)
0927 05:09:45.348783 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.348802 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:45.348965 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:45.349339 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:45.349346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:45.349372 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:45.349792 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.349801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.351381 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.351386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.354270 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.354286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.356039 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.356046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.356303 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.356402 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 05:09:45.356506 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:09:45.356671 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":311,"thread_start_us":917,"threads_started":1}
I20260927 05:09:45.357194 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.347831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38555 (local address 127.0.97.1:57397)
0927 05:09:45.348673 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.348695 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:09:45.348873 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:09:45.349294 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:09:45.349300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:09:45.349331 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:09:45.350000 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.350015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.351383 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.351388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.354270 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:09:45.354286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.356051 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.356057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:09:45.356301 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.356401 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 05:09:45.356506 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 05:09:45.356671 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":283,"thread_start_us":666,"threads_started":1}
I20260927 05:09:45.358249 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256e623ed7aa4640a612e6b005431d8e" candidate_uuid: "00d4ba2781e54dc2955557054383c27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ef1e853a264b39a83fc939de8ef703" is_pre_election: true
I20260927 05:09:45.358352 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256e623ed7aa4640a612e6b005431d8e" candidate_uuid: "00d4ba2781e54dc2955557054383c27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a928f8d66224576be6e2d8caf31ce04" is_pre_election: true
I20260927 05:09:45.358855 16215 raft_consensus.cc:2454] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d4ba2781e54dc2955557054383c27d in term 0.
I20260927 05:09:45.358855 16349 raft_consensus.cc:2454] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d4ba2781e54dc2955557054383c27d in term 0.
I20260927 05:09:45.359591 16017 leader_election.cc:302] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d4ba2781e54dc2955557054383c27d, 68ef1e853a264b39a83fc939de8ef703; no voters:
I20260927 05:09:45.360105 16419 raft_consensus.cc:2781] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:09:45.360244 16419 raft_consensus.cc:482] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:09:45.360392 16419 raft_consensus.cc:3037] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:45.362677 16419 raft_consensus.cc:504] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } }
I20260927 05:09:45.363365 16419 leader_election.cc:288] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 1 election: Requested vote from peers 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555), 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
I20260927 05:09:45.363806 16349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256e623ed7aa4640a612e6b005431d8e" candidate_uuid: "00d4ba2781e54dc2955557054383c27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a928f8d66224576be6e2d8caf31ce04"
I20260927 05:09:45.363837 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256e623ed7aa4640a612e6b005431d8e" candidate_uuid: "00d4ba2781e54dc2955557054383c27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ef1e853a264b39a83fc939de8ef703"
I20260927 05:09:45.364050 16349 raft_consensus.cc:3037] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:45.364077 16215 raft_consensus.cc:3037] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:09:45.366148 16215 raft_consensus.cc:2454] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d4ba2781e54dc2955557054383c27d in term 1.
I20260927 05:09:45.366214 16349 raft_consensus.cc:2454] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d4ba2781e54dc2955557054383c27d in term 1.
I20260927 05:09:45.366597 16017 leader_election.cc:302] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d4ba2781e54dc2955557054383c27d, 68ef1e853a264b39a83fc939de8ef703; no voters:
I20260927 05:09:45.366879 16419 raft_consensus.cc:2781] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:09:45.367367 16419 raft_consensus.cc:686] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 1 LEADER]: Becoming Leader. State: Replica: 00d4ba2781e54dc2955557054383c27d, State: Running, Role: LEADER
I20260927 05:09:45.367851 16419 consensus_queue.cc:237] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } }
I20260927 05:09:45.372485 15936 catalog_manager.cc:5495] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d reported cstate change: term changed from 0 to 1, leader changed from <none> to 00d4ba2781e54dc2955557054383c27d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "00d4ba2781e54dc2955557054383c27d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } health_report { overall_health: HEALTHY } } }
I20260927 05:09:45.429816 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.424564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.0.97.1:33473)
0927 05:09:45.424795 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.424799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.424816 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:09:45.424905 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.424909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.424985 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.425116 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:09:45.425490 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.426256 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.427705 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.428202 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.428305 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.428823 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:09:45.429346 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:09:45.429495 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.429535 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.429606 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:09:45.438462 16215 raft_consensus.cc:1261] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703 [term 1 FOLLOWER]: Refusing update from remote peer 00d4ba2781e54dc2955557054383c27d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:09:45.438460 16349 raft_consensus.cc:1261] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04 [term 1 FOLLOWER]: Refusing update from remote peer 00d4ba2781e54dc2955557054383c27d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:09:45.439489 16419 consensus_queue.cc:1035] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:09:45.439821 16424 consensus_queue.cc:1035] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:09:45.452757 16427 mvcc.cc:205] Tried to move back new op lower bound from 7333829777145200640 to 7333829776870436864. Current Snapshot: MvccSnapshot[applied={T|T < 7333829777145200640}]
I20260927 05:09:45.468187 15937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:09:45.468488 15937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:09:45.472348 15937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Sending DeleteTablet for 3 replicas of tablet 256e623ed7aa4640a612e6b005431d8e
I20260927 05:09:45.473685 16195 tablet_service.cc:1514] Processing DeleteTablet for tablet 256e623ed7aa4640a612e6b005431d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57518
I20260927 05:09:45.473701 16061 tablet_service.cc:1514] Processing DeleteTablet for tablet 256e623ed7aa4640a612e6b005431d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34330
I20260927 05:09:45.474355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 15998
I20260927 05:09:45.474525 16329 tablet_service.cc:1514] Processing DeleteTablet for tablet 256e623ed7aa4640a612e6b005431d8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:09:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53148
I20260927 05:09:45.480630 16437 tablet_replica.cc:330] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: stopping tablet replica
I20260927 05:09:45.481359 16437 raft_consensus.cc:2229] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:09:45.487201 16439 tablet_replica.cc:330] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: stopping tablet replica
I20260927 05:09:45.487407 16437 raft_consensus.cc:2258] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:45.488281 16439 raft_consensus.cc:2229] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:09:45.488454 16438 tablet_replica.cc:330] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: stopping tablet replica
I20260927 05:09:45.488796 16439 raft_consensus.cc:2258] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:45.488895 16438 raft_consensus.cc:2229] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:09:45.489617 16438 raft_consensus.cc:2258] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:09:45.490777 16439 ts_tablet_manager.cc:1905] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:09:45.491310 16437 ts_tablet_manager.cc:1905] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 05:09:45 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:09:45.496730 16439 ts_tablet_manager.cc:1918] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:09:45.496912 16439 log.cc:1196] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/256e623ed7aa4640a612e6b005431d8e
I20260927 05:09:45.497462 16439 ts_tablet_manager.cc:1939] T 256e623ed7aa4640a612e6b005431d8e P 9a928f8d66224576be6e2d8caf31ce04: Deleting consensus metadata
I20260927 05:09:45.498551 16437 ts_tablet_manager.cc:1918] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:09:45.498790 16437 log.cc:1196] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/256e623ed7aa4640a612e6b005431d8e
I20260927 05:09:45.499379 16437 ts_tablet_manager.cc:1939] T 256e623ed7aa4640a612e6b005431d8e P 68ef1e853a264b39a83fc939de8ef703: Deleting consensus metadata
I20260927 05:09:45.499504 15922 catalog_manager.cc:4841] TS 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555): tablet 256e623ed7aa4640a612e6b005431d8e (table test-table [id=cf1370982e4641eeb15bbffd5de8e21c]) successfully deleted
I20260927 05:09:45.501296 15924 catalog_manager.cc:4841] TS 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465): tablet 256e623ed7aa4640a612e6b005431d8e (table test-table [id=cf1370982e4641eeb15bbffd5de8e21c]) successfully deleted
I20260927 05:09:45.501893 16438 ts_tablet_manager.cc:1905] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:09:45.505946 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.475520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.0.97.1:33473)
0927 05:09:45.475695 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.475769 (+ 74us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.484931 (+ 9162us) server_negotiation.cc:407] Connection header received
0927 05:09:45.485457 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.485464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.485550 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.485638 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:45.487880 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.489156 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.491190 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.491834 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.492002 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.497951 (+ 5949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:45.497969 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:45.497988 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:45.498040 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:45.502382 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:45.503388 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:45.503398 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:45.503401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:45.503575 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:45.504027 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:45.504033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:45.504036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:45.504532 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:45.504834 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.505298 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.505557 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:09:45.506672 16121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:45.606611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 15998
I20260927 05:09:45.624114 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 16132
W20260927 05:09:45.624184 15921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33473 (error 108)
I20260927 05:09:45.625969 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.625562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:34360)
0927 05:09:45.625726 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.625823 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 05:09:45.626179 15921 catalog_manager.cc:4584] TS 00d4ba2781e54dc2955557054383c27d (127.0.97.1:33473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 256e623ed7aa4640a612e6b005431d8e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Sep 27 05:09:45 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:09:45.640264 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.624902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.0.97.2:44465)
0927 05:09:45.625081 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.625087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.625216 (+ 129us) server_negotiation.cc:407] Connection header received
0927 05:09:45.625435 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.625445 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.625534 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.625642 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:45.627160 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.627890 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.629341 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.629735 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.629885 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.634269 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:45.634284 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:45.634301 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:45.634339 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:45.637403 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:45.638099 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:45.638107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:45.638109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:45.638232 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:45.638610 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:45.638616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:45.638619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:45.639012 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:45.639306 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.639652 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.639904 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:09:45.640920 16255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:45.680002 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.679451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:34376)
0927 05:09:45.679680 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.679777 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 05:09:45.681046 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.680689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:34388)
0927 05:09:45.680840 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.680929 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 05:09:45.732183 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 16132
I20260927 05:09:45.741534 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.740939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:34390)
0927 05:09:45.741194 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.741318 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 05:09:45.742729 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.742272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:34396)
0927 05:09:45.742462 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.742559 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 05:09:45.750003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 16266
Sep 27 05:09:45 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:09:45.765781 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.750764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.0.97.3:38555)
0927 05:09:45.750964 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 05:09:45.750970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:09:45.750990 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:09:45.751167 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:09:45.751173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:09:45.751265 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:09:45.751370 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:09:45.752751 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.753473 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.754787 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:09:45.755183 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:09:45.755328 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:09:45.760103 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:09:45.760119 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:09:45.760136 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:09:45.760173 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:09:45.762743 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:45.763448 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:45.763456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:45.763458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:45.763608 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:09:45.764257 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:09:45.764260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:09:45.764262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:09:45.764600 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:09:45.764899 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 05:09:45.765231 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 05:09:45.765452 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:09:45.766474 16389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:45.823577 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.822912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:34404)
0927 05:09:45.823189 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.823315 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 05:09:45.824908 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:09:45.824419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:34412)
0927 05:09:45.824645 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 05:09:45.824757 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 05:09:45.857398 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 16266
I20260927 05:09:45.877238 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 15905
I20260927 05:09:45.878440 15967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:09:45.990468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 15905
I20260927 05:10:06.012348 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34651
--webserver_interface=127.0.97.62
--webserver_port=36865
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:06.354842 16446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:06.355160 16446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:06.355330 16446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:06.362705 16446 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 05:10:06.362812 16446 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 05:10:06.362851 16446 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 05:10:06.362967 16446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:10:06.363037 16446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:06.363160 16446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:06.363224 16446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:10:06.363332 16446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:10:06.373361 16446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:06.374751 16446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:06.376166 16446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:06.400416 16455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:06.400468 16453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:06.400374 16452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:06.405809 16446 server_base.cc:1034] running on GCE node
I20260927 05:10:06.406857 16446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:06.408715 16446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:06.410002 16446 hybrid_clock.cc:648] HybridClock initialized: now 1790485806409937 us; error 46 us; skew 500 ppm
I20260927 05:10:06.410501 16446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:06 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:06.419510 16446 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:10:06.421542 16446 webserver.cc:460] Webserver started at http://127.0.97.62:36865/ using document root <none> and password file <none>
I20260927 05:10:06.422441 16446 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:06.422597 16446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:06.428826 16446 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 05:10:06.432049 16462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:06.433821 16446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:10:06.434058 16446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "5587b12ee129401e83435f9cd8c84429"
format_stamp: "Formatted at 2026-09-27 05:09:43 on dist-test-slave-hgs0"
server_key: "6f911aebbc646d1dd45dd6e967cb51dc"
server_key_iv: "3e315f95f7d72a853bc237cb19e9ac6d"
server_key_version: "encryptionkey@0"
I20260927 05:10:06.435218 16446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:06.455267 16446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:06.459851 16446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:06.460198 16446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:06.480314 16446 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34651
I20260927 05:10:06.480337 16513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34651 every 8 connection(s)
I20260927 05:10:06.481967 16446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:10:06.487179 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 16446
I20260927 05:10:06.488224 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33473
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34651
--webserver_port=45937
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:10:06.490607 16514 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:10:06.496163 16514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Bootstrap starting.
I20260927 05:10:06.516963 16514 log.cc:824] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:06.529155 16514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:10:06.529736 16514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Bootstrap complete.
I20260927 05:10:06.539438 16514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:10:06.540406 16514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5587b12ee129401e83435f9cd8c84429, State: Initialized, Role: FOLLOWER
I20260927 05:10:06.541049 16514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:10:06.541290 16514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:10:06.541414 16514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:10:06.541606 16514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 1 FOLLOWER]: Advancing to term 2
I20260927 05:10:06.545648 16514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:10:06.546204 16514 leader_election.cc:302] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5587b12ee129401e83435f9cd8c84429; no voters:
I20260927 05:10:06.546887 16514 leader_election.cc:288] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 05:10:06.547030 16518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 05:10:06.548385 16518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 2 LEADER]: Becoming Leader. State: Replica: 5587b12ee129401e83435f9cd8c84429, State: Running, Role: LEADER
I20260927 05:10:06.549017 16518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:10:06.549335 16514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:10:06.551721 16520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5587b12ee129401e83435f9cd8c84429. Latest consensus state: current_term: 2 leader_uuid: "5587b12ee129401e83435f9cd8c84429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } } }
I20260927 05:10:06.552102 16520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:06.552596 16519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5587b12ee129401e83435f9cd8c84429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } } }
I20260927 05:10:06.552982 16519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:06.561477 16523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:10:06.570468 16523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:10:06.572072 16523 catalog_manager.cc:1259] Loaded cluster ID: 123b2d849b7c4a0eb9741b94d6fc19dc
I20260927 05:10:06.572242 16523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:10:06.576524 16523 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:10:06.578148 16523 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Loaded TSK: 0
I20260927 05:10:06.610437 16523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Generated new TSK 1
I20260927 05:10:06.610872 16523 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 05:10:06.854501 16516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:06.854835 16516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:06.854996 16516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:06.862797 16516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:06.862959 16516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:06.863168 16516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:10:06.873725 16516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45937
--tserver_master_addrs=127.0.97.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:06.875160 16516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:06.876581 16516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:06.891614 16542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:06.891614 16541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:06.892398 16544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:06.892760 16516 server_base.cc:1034] running on GCE node
I20260927 05:10:06.893440 16516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:06.894492 16516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:06.895767 16516 hybrid_clock.cc:648] HybridClock initialized: now 1790485806895677 us; error 80 us; skew 500 ppm
I20260927 05:10:06.896183 16516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:06 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:06.899441 16516 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:10:06.901149 16516 webserver.cc:460] Webserver started at http://127.0.97.1:45937/ using document root <none> and password file <none>
I20260927 05:10:06.901863 16516 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:06.902012 16516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:06.907873 16516 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 05:10:06.911181 16551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:06.912772 16516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:10:06.912953 16516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "00d4ba2781e54dc2955557054383c27d"
format_stamp: "Formatted at 2026-09-27 05:09:43 on dist-test-slave-hgs0"
server_key: "d39a9ff92fe0505df159b031081810e2"
server_key_iv: "e68cd55d198bba07f0856f2caa3eb951"
server_key_version: "encryptionkey@0"
I20260927 05:10:06.913949 16516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:06.933519 16516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:06.938216 16516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:06.938517 16516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:06.939695 16516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:06.942004 16558 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 05:10:06.948509 16558 ts_tablet_manager.cc:1852] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 05:10:06.949498 16558 ts_tablet_manager.cc:1905] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:06.956866 16558 ts_tablet_manager.cc:1918] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:06.957077 16558 log.cc:1196] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/256e623ed7aa4640a612e6b005431d8e
I20260927 05:10:06.957569 16558 ts_tablet_manager.cc:1939] T 256e623ed7aa4640a612e6b005431d8e P 00d4ba2781e54dc2955557054383c27d: Deleting consensus metadata
I20260927 05:10:06.957976 16516 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 05:10:06.958132 16516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260927 05:10:06.958277 16516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:06.958369 16516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:06.993216 16516 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33473
I20260927 05:10:06.993270 16664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33473 every 8 connection(s)
I20260927 05:10:06.995045 16516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:10:06.996846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 16516
I20260927 05:10:06.997571 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44465
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34651
--webserver_port=46093
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:10:07 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:07.028074 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:06.998266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56187 (local address 127.0.97.62:34651)
0927 05:10:07.004816 (+ 6550us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:07.004835 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:07.004895 (+ 60us) server_negotiation.cc:407] Connection header received
0927 05:10:07.005343 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:07.005374 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:07.005868 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:07.006300 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:07.008965 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.011150 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.013731 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.014868 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.015063 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:07.020171 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:07.020179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:07.020195 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:07.020238 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:07.023270 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:07.024369 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:07.024377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:07.024382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:07.024542 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:07.025039 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:07.025042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:07.025044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:07.025512 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:07.025663 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:07.026694 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0927 05:10:07.027044 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6262,"thread_start_us":6103,"threads_started":1}
I20260927 05:10:07.028270 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:06.998559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:56187)
0927 05:10:06.999626 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:06.999692 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:07.003306 (+ 3614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:07.006590 (+ 3284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:07.006602 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:07.007627 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:07.008673 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:07.008699 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.011357 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.011363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:07.013531 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:07.013548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.015482 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.015489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:07.015773 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:07.016648 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:07.016682 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:07.019798 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:07.023432 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:07.023445 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:07.023465 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:07.024215 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:07.024667 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:07.024675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:07.024680 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:07.024921 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:07.025666 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:07.025676 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:07.026152 (+ 476us) client_negotiation.cc:769] Sending connection context
0927 05:10:07.026494 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 05:10:07.027032 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":373,"threads_started":1}
I20260927 05:10:07.031442 16665 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34651
I20260927 05:10:07.031778 16665 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:07.032603 16665 heartbeater.cc:511] Master 127.0.97.62:34651 requested a full tablet report, sending...
I20260927 05:10:07.035116 16479 ts_manager.cc:194] Registered new tserver with Master: 00d4ba2781e54dc2955557054383c27d (127.0.97.1:33473)
I20260927 05:10:07.037129 16479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56187
W20260927 05:10:07.348448 16669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:07.348734 16669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:07.348829 16669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:07.356315 16669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:07.356421 16669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:07.356537 16669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:10:07.366446 16669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46093
--tserver_master_addrs=127.0.97.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:07.367728 16669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:07.369037 16669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:07.383116 16675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:07.383087 16674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:07.383328 16677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:07.383682 16669 server_base.cc:1034] running on GCE node
I20260927 05:10:07.384563 16669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:07.385717 16669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:07.386952 16669 hybrid_clock.cc:648] HybridClock initialized: now 1790485807386901 us; error 55 us; skew 500 ppm
I20260927 05:10:07.387320 16669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:07 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:07.390735 16669 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:10:07.392455 16669 webserver.cc:460] Webserver started at http://127.0.97.2:46093/ using document root <none> and password file <none>
I20260927 05:10:07.393141 16669 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:07.393260 16669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:07.399137 16669 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:10:07.402149 16684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:07.403868 16669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:10:07.404127 16669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "68ef1e853a264b39a83fc939de8ef703"
format_stamp: "Formatted at 2026-09-27 05:09:44 on dist-test-slave-hgs0"
server_key: "207b750ad9af11bb902737205bed0edb"
server_key_iv: "e28d54eac57cb590c0bec82aceb7c4e2"
server_key_version: "encryptionkey@0"
I20260927 05:10:07.405150 16669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:07.430406 16669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:07.435590 16669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:07.435856 16669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:07.437016 16669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:07.438426 16669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:07.438524 16669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:07.438643 16669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:07.438737 16669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:07.475836 16669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44465
I20260927 05:10:07.475941 16796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44465 every 8 connection(s)
I20260927 05:10:07.477459 16669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:10:07.487780 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 16669
I20260927 05:10:07.488523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38555
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34651
--webserver_port=43289
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 05:10:07 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:07.503357 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:07.480441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42697 (local address 127.0.97.62:34651)
0927 05:10:07.480655 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:07.480662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:07.481912 (+ 1250us) server_negotiation.cc:407] Connection header received
0927 05:10:07.483828 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:07.483834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:07.483916 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:07.484056 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:07.485738 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.486822 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.489307 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.490024 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.490153 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:07.495203 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:07.495214 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:07.495222 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:07.495280 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:07.499315 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:07.500523 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:07.500533 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:07.500539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:07.500739 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:07.501238 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:07.501246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:07.501252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:07.501670 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:07.501839 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:07.502591 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 05:10:07.502893 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:10:07.504217 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:07.480904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:42697)
0927 05:10:07.481730 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:07.481774 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:07.483540 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:07.484298 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:07.484306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:07.484829 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:07.485533 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:07.485562 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.487023 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.487027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:07.489123 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:07.489143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.490304 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.490314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:07.490663 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:07.491527 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:07.491564 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:07.494859 (+ 3295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:07.499663 (+ 4804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:07.499673 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:07.499712 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:07.500357 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:07.500873 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:07.500876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:07.500878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:07.501085 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:07.501809 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:07.501815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:07.502173 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 05:10:07.502433 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 05:10:07.502895 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":363,"threads_started":1}
I20260927 05:10:07.505944 16797 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34651
I20260927 05:10:07.506342 16797 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:07.507134 16797 heartbeater.cc:511] Master 127.0.97.62:34651 requested a full tablet report, sending...
I20260927 05:10:07.508976 16479 ts_manager.cc:194] Registered new tserver with Master: 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
I20260927 05:10:07.510004 16479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42697
W20260927 05:10:07.830169 16800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:07.830482 16800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:07.830651 16800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:07.838299 16800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:07.838421 16800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:07.838634 16800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:10:07.849012 16800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43289
--tserver_master_addrs=127.0.97.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:07.850481 16800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:07.851878 16800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:07.866901 16805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:07.866933 16808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:07.866955 16806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:07.868104 16800 server_base.cc:1034] running on GCE node
I20260927 05:10:07.868810 16800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:07.869863 16800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:07.871101 16800 hybrid_clock.cc:648] HybridClock initialized: now 1790485807871041 us; error 56 us; skew 500 ppm
I20260927 05:10:07.871438 16800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:07 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:07.874671 16800 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:10:07.876240 16800 webserver.cc:460] Webserver started at http://127.0.97.3:43289/ using document root <none> and password file <none>
I20260927 05:10:07.876906 16800 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:07.877017 16800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:07.882525 16800 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:10:07.885537 16815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:07.887189 16800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:10:07.887382 16800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "9a928f8d66224576be6e2d8caf31ce04"
format_stamp: "Formatted at 2026-09-27 05:09:45 on dist-test-slave-hgs0"
server_key: "d6649204c886593805eb58290581e2e3"
server_key_iv: "33c608f44d25cc47975a38a590c00c8d"
server_key_version: "encryptionkey@0"
I20260927 05:10:07.888587 16800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:07.908367 16800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:07.912948 16800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:07.913254 16800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:07.914286 16800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:07.915725 16800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:07.915822 16800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:07.915940 16800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:07.916054 16800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 05:10:07.951903 16800 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38555
I20260927 05:10:07.952014 16927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38555 every 8 connection(s)
I20260927 05:10:07.953500 16800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:10:07.957667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 16800
Sep 27 05:10:07 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:07.976747 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:07.958597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.0.97.62:34651)
0927 05:10:07.960473 (+ 1876us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:07.960482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:07.961048 (+ 566us) server_negotiation.cc:407] Connection header received
0927 05:10:07.961251 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:07.961259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:07.961320 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:07.961435 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:07.963234 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.964232 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.965770 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.966311 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.966453 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:07.969212 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:07.969217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:07.969220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:07.969252 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:07.973792 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:07.974521 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:07.974527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:07.974531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:07.974679 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:07.975085 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:07.975092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:07.975097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:07.975396 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:07.975707 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:07.976045 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 05:10:07.976284 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":1711,"thread_start_us":1572,"threads_started":1}
I20260927 05:10:07.977146 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:07.956238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60929 (local address 127.0.97.62:34651)
0927 05:10:07.956435 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:07.956439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:07.957534 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 05:10:07.960432 (+ 2898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:07.960437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:07.960533 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:07.960652 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:07.962525 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.963556 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.965432 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.966060 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.966161 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:07.970016 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:07.970021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:07.970024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:07.970063 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:07.973723 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:07.974583 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:07.974591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:07.974594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:07.974765 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:07.975248 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:07.975255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:07.975263 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:07.975535 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:07.975706 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:07.976510 (+ 804us) server_negotiation.cc:299] Negotiation successful
0927 05:10:07.976736 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:10:07.978875 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:07.956635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:60929)
0927 05:10:07.957366 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:07.957412 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:07.960074 (+ 2662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:07.960927 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:07.960936 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:07.961613 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:07.962299 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:07.962334 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.963794 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.963800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:07.965294 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:07.965307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.966299 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.966306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:07.966652 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:07.967444 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:07.967467 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:07.969815 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:07.973911 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:07.973919 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:07.973935 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:07.974423 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:07.974897 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:07.974903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:07.974907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:07.975111 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:07.975673 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:07.975682 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:07.976115 (+ 433us) client_negotiation.cc:769] Sending connection context
0927 05:10:07.976405 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 05:10:07.977595 (+ 1190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":286,"threads_started":1}
I20260927 05:10:07.980340 16928 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34651
I20260927 05:10:07.980675 16928 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:07.981449 16928 heartbeater.cc:511] Master 127.0.97.62:34651 requested a full tablet report, sending...
I20260927 05:10:07.982893 16479 ts_manager.cc:194] Registered new tserver with Master: 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555)
I20260927 05:10:07.983777 16479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60929
I20260927 05:10:07.992350 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:10:08.000314 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:07.993480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.0.97.62:34651)
0927 05:10:07.993749 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:07.993759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:07.994189 (+ 430us) server_negotiation.cc:407] Connection header received
0927 05:10:07.994487 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:07.994492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:07.994649 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:07.994782 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:10:07.995306 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.996191 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.998343 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:07.998772 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:07.998869 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:07.999460 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:10:07.999703 (+ 243us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 05:10:08.000084 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60184: token expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 05:10:08.000466 16667 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60184: token expired
W20260927 05:10:08.001080 16398 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:34651)
I20260927 05:10:08.014951 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.001533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.0.97.62:34651)
0927 05:10:08.001697 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:08.001703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:08.001720 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:10:08.001883 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:08.001892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:08.001963 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:08.002077 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:08.003412 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.004210 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.005522 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.005935 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.006082 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.009052 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:08.009057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:08.009060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:08.009098 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:08.012158 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:08.012902 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:08.012913 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:08.012916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:08.013040 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:08.013442 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:08.013447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:08.013451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:08.013782 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:08.014050 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:08.014385 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 05:10:08.014628 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 05:10:08.018926 16479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:10:08.026543 16479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:10:08.041050 16665 heartbeater.cc:503] Master 127.0.97.62:34651 was elected leader, sending a full tablet report...
I20260927 05:10:08.063652 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.042345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.0.97.2:44465)
0927 05:10:08.053971 (+ 11626us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:08.053979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:08.055470 (+ 1491us) server_negotiation.cc:407] Connection header received
0927 05:10:08.055577 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:08.055585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:08.055824 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:08.056078 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:08.056588 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.058090 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.060709 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.062147 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.062303 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.062557 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:08.063125 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 05:10:08.063307 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11467,"thread_start_us":11316,"threads_started":1}
I20260927 05:10:08.063717 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.041679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44465 (local address 127.0.0.1:36158)
0927 05:10:08.055193 (+ 13514us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.055217 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:08.055341 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:08.056079 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:08.056084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:08.056116 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:08.056428 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:08.056437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.058227 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.058235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:08.060484 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:08.060500 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.062406 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.062413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:08.062649 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.062715 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 05:10:08.062866 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 05:10:08.063240 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13378,"mutex_wait_us":202,"thread_start_us":13201,"threads_started":1}
I20260927 05:10:08.064320 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.039569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.0.97.1:33473)
0927 05:10:08.053726 (+ 14157us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:08.053736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:08.053766 (+ 30us) server_negotiation.cc:407] Connection header received
0927 05:10:08.053868 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:08.053877 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:08.055233 (+ 1356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:08.055428 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:08.056225 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.057706 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.060877 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.062444 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.062576 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.062783 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:08.063492 (+ 709us) server_negotiation.cc:299] Negotiation successful
0927 05:10:08.063680 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13985,"thread_start_us":13863,"threads_started":1}
I20260927 05:10:08.064348 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.039407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.0.97.3:38555)
0927 05:10:08.054311 (+ 14904us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:08.054321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:08.054359 (+ 38us) server_negotiation.cc:407] Connection header received
0927 05:10:08.054459 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:08.054466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:08.054745 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:08.055693 (+ 948us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:08.056014 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.058111 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.060876 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.062602 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.062799 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.063029 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:08.063812 (+ 783us) server_negotiation.cc:299] Negotiation successful
0927 05:10:08.064022 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14729,"thread_start_us":13791,"threads_started":1}
I20260927 05:10:08.064455 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.039212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:42780)
0927 05:10:08.052750 (+ 13538us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.052774 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:08.052918 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:08.055492 (+ 2574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:08.055496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:08.055529 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:08.055978 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:08.055992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.057882 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.057887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:08.060669 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:08.060690 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.062961 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.062968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:08.063155 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.063280 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 05:10:08.063355 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 05:10:08.064080 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13342,"mutex_wait_us":673,"thread_start_us":12203,"threads_started":1}
I20260927 05:10:08.064885 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.039126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38555 (local address 127.0.0.1:60766)
0927 05:10:08.050816 (+ 11690us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.050891 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:08.051152 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:08.055042 (+ 3890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:08.055056 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:08.055096 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:08.055570 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:08.055586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.058269 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.058279 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:08.060689 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:08.060699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.062910 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.062915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:08.063140 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.063275 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 05:10:08.063363 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 05:10:08.063514 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11291,"mutex_wait_us":102,"thread_start_us":13280,"threads_started":1}
I20260927 05:10:08.068032 16732 tablet_service.cc:1467] Processing CreateTablet for tablet fb0b9c2bc2bd4b21bacd789082c49055 (DEFAULT_TABLE table=test-table [id=a78e23abb3b54e21a368e514847dd71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:08.068521 16863 tablet_service.cc:1467] Processing CreateTablet for tablet fb0b9c2bc2bd4b21bacd789082c49055 (DEFAULT_TABLE table=test-table [id=a78e23abb3b54e21a368e514847dd71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:08.069453 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb0b9c2bc2bd4b21bacd789082c49055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:08.069042 16600 tablet_service.cc:1467] Processing CreateTablet for tablet fb0b9c2bc2bd4b21bacd789082c49055 (DEFAULT_TABLE table=test-table [id=a78e23abb3b54e21a368e514847dd71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:08.069990 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb0b9c2bc2bd4b21bacd789082c49055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:08.070232 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb0b9c2bc2bd4b21bacd789082c49055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:08.080369 16944 tablet_bootstrap.cc:492] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Bootstrap starting.
I20260927 05:10:08.083209 16945 tablet_bootstrap.cc:492] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: Bootstrap starting.
I20260927 05:10:08.084427 16944 tablet_bootstrap.cc:654] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:08.085501 16944 log.cc:824] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:08.087852 16945 tablet_bootstrap.cc:654] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:08.089159 16945 log.cc:824] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:08.089872 16944 tablet_bootstrap.cc:492] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: No bootstrap required, opened a new log
I20260927 05:10:08.090119 16946 tablet_bootstrap.cc:492] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: Bootstrap starting.
I20260927 05:10:08.090160 16944 ts_tablet_manager.cc:1397] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 05:10:08.092527 16945 tablet_bootstrap.cc:492] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: No bootstrap required, opened a new log
I20260927 05:10:08.092880 16945 ts_tablet_manager.cc:1397] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 05:10:08.093322 16946 tablet_bootstrap.cc:654] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:08.094444 16946 log.cc:824] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:08.098642 16946 tablet_bootstrap.cc:492] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: No bootstrap required, opened a new log
I20260927 05:10:08.098927 16946 ts_tablet_manager.cc:1397] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260927 05:10:08.101420 16944 raft_consensus.cc:348] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:08.101898 16944 raft_consensus.cc:374] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:08.102051 16944 raft_consensus.cc:729] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d4ba2781e54dc2955557054383c27d, State: Initialized, Role: FOLLOWER
I20260927 05:10:08.102766 16944 consensus_queue.cc:260] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:08.103606 16945 raft_consensus.cc:348] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:08.104105 16945 raft_consensus.cc:374] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:08.104241 16945 raft_consensus.cc:729] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ef1e853a264b39a83fc939de8ef703, State: Initialized, Role: FOLLOWER
I20260927 05:10:08.104869 16944 ts_tablet_manager.cc:1428] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 05:10:08.104889 16945 consensus_queue.cc:260] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:08.106717 16945 ts_tablet_manager.cc:1428] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 05:10:08.106813 16797 heartbeater.cc:503] Master 127.0.97.62:34651 was elected leader, sending a full tablet report...
I20260927 05:10:08.108294 16946 raft_consensus.cc:348] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:08.108774 16946 raft_consensus.cc:374] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:08.108883 16946 raft_consensus.cc:729] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a928f8d66224576be6e2d8caf31ce04, State: Initialized, Role: FOLLOWER
I20260927 05:10:08.109673 16946 consensus_queue.cc:260] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:08.111123 16928 heartbeater.cc:503] Master 127.0.97.62:34651 was elected leader, sending a full tablet report...
I20260927 05:10:08.111758 16946 ts_tablet_manager.cc:1428] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260927 05:10:08.125236 16950 raft_consensus.cc:482] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:10:08.125504 16950 raft_consensus.cc:504] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:08.127152 16950 leader_election.cc:288] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555), 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
I20260927 05:10:08.134446 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.127489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34455 (local address 127.0.97.3:38555)
0927 05:10:08.127827 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:08.127836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:08.128669 (+ 833us) server_negotiation.cc:407] Connection header received
0927 05:10:08.128862 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:08.128867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:08.128949 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:08.129037 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:08.129723 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.130503 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.132559 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.133352 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.133501 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.133609 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:08.134010 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 05:10:08.134149 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260927 05:10:08.134485 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.127616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37859 (local address 127.0.97.2:44465)
0927 05:10:08.127854 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:08.127862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:08.128846 (+ 984us) server_negotiation.cc:407] Connection header received
0927 05:10:08.129032 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:08.129039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:08.129116 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:08.129236 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:08.129890 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.130608 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.132540 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.133512 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.133649 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.133749 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:08.134080 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 05:10:08.134224 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 05:10:08.134769 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.127623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44465 (local address 127.0.97.1:37859)
0927 05:10:08.128694 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.128716 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:08.128872 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:08.129272 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:08.129276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:08.129297 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:08.129703 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:08.129715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.130759 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.130763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:08.132380 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:08.132392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.133749 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.133753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:08.133880 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.133903 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 05:10:08.134030 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 05:10:08.134155 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":3,"thread_start_us":607,"threads_started":1}
I20260927 05:10:08.134994 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.127551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38555 (local address 127.0.97.1:34455)
0927 05:10:08.128543 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.128559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:08.128687 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:08.129073 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:08.129077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:08.129102 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:08.129549 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:08.129562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.130644 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.130652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:08.132377 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:08.132390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.133618 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.133624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:08.133827 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.133855 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 05:10:08.133935 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 05:10:08.134081 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":32,"thread_start_us":637,"threads_started":1}
I20260927 05:10:08.135996 16883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0b9c2bc2bd4b21bacd789082c49055" candidate_uuid: "00d4ba2781e54dc2955557054383c27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a928f8d66224576be6e2d8caf31ce04" is_pre_election: true
I20260927 05:10:08.135996 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0b9c2bc2bd4b21bacd789082c49055" candidate_uuid: "00d4ba2781e54dc2955557054383c27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ef1e853a264b39a83fc939de8ef703" is_pre_election: true
I20260927 05:10:08.136597 16752 raft_consensus.cc:2454] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d4ba2781e54dc2955557054383c27d in term 0.
I20260927 05:10:08.136597 16883 raft_consensus.cc:2454] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d4ba2781e54dc2955557054383c27d in term 0.
I20260927 05:10:08.137363 16553 leader_election.cc:302] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d4ba2781e54dc2955557054383c27d, 9a928f8d66224576be6e2d8caf31ce04; no voters:
I20260927 05:10:08.137874 16950 raft_consensus.cc:2781] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:10:08.138048 16950 raft_consensus.cc:482] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:10:08.138167 16950 raft_consensus.cc:3037] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:08.140611 16950 raft_consensus.cc:504] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:08.141273 16950 leader_election.cc:288] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 1 election: Requested vote from peers 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555), 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
I20260927 05:10:08.141811 16752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0b9c2bc2bd4b21bacd789082c49055" candidate_uuid: "00d4ba2781e54dc2955557054383c27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ef1e853a264b39a83fc939de8ef703"
I20260927 05:10:08.141811 16883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0b9c2bc2bd4b21bacd789082c49055" candidate_uuid: "00d4ba2781e54dc2955557054383c27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a928f8d66224576be6e2d8caf31ce04"
I20260927 05:10:08.142045 16883 raft_consensus.cc:3037] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:08.142045 16752 raft_consensus.cc:3037] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:08.145488 16883 raft_consensus.cc:2454] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d4ba2781e54dc2955557054383c27d in term 1.
I20260927 05:10:08.145488 16752 raft_consensus.cc:2454] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d4ba2781e54dc2955557054383c27d in term 1.
I20260927 05:10:08.145947 16553 leader_election.cc:302] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d4ba2781e54dc2955557054383c27d, 9a928f8d66224576be6e2d8caf31ce04; no voters:
I20260927 05:10:08.146250 16950 raft_consensus.cc:2781] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:10:08.146695 16950 raft_consensus.cc:686] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 LEADER]: Becoming Leader. State: Replica: 00d4ba2781e54dc2955557054383c27d, State: Running, Role: LEADER
I20260927 05:10:08.147218 16950 consensus_queue.cc:237] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:08.152515 16479 catalog_manager.cc:5495] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d reported cstate change: term changed from 0 to 1, leader changed from <none> to 00d4ba2781e54dc2955557054383c27d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "00d4ba2781e54dc2955557054383c27d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } health_report { overall_health: UNKNOWN } } }
I20260927 05:10:08.169787 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.163902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.0.97.1:33473)
0927 05:10:08.164098 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:08.164103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:08.164123 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:10:08.164342 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:08.164346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:08.164432 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:08.164568 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:10:08.164919 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.165685 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.167252 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.167760 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.167894 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.168558 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:10:08.169201 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:10:08.169381 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:08.169431 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 05:10:08.169519 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 05:10:08.179378 16752 raft_consensus.cc:1261] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703 [term 1 FOLLOWER]: Refusing update from remote peer 00d4ba2781e54dc2955557054383c27d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:10:08.179380 16883 raft_consensus.cc:1261] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04 [term 1 FOLLOWER]: Refusing update from remote peer 00d4ba2781e54dc2955557054383c27d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:10:08.180476 16955 consensus_queue.cc:1035] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:10:08.180886 16950 consensus_queue.cc:1035] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:10:08.193594 16957 mvcc.cc:205] Tried to move back new op lower bound from 7333829870291501056 to 7333829870176542720. Current Snapshot: MvccSnapshot[applied={T|T < 7333829870291501056}]
I20260927 05:10:08.194968 16959 mvcc.cc:205] Tried to move back new op lower bound from 7333829870291501056 to 7333829870176542720. Current Snapshot: MvccSnapshot[applied={T|T < 7333829870291501056}]
I20260927 05:10:08.216265 16478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:10:08.216632 16478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:10:08.220935 16478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Sending DeleteTablet for 3 replicas of tablet fb0b9c2bc2bd4b21bacd789082c49055
I20260927 05:10:08.222419 16863 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0b9c2bc2bd4b21bacd789082c49055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60766
I20260927 05:10:08.222450 16732 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0b9c2bc2bd4b21bacd789082c49055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36158
I20260927 05:10:08.222879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 16516
I20260927 05:10:08.223523 16600 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0b9c2bc2bd4b21bacd789082c49055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42780
I20260927 05:10:08.249567 16968 tablet_replica.cc:330] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: stopping tablet replica
I20260927 05:10:08.247385 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.224714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.0.97.1:33473)
0927 05:10:08.224932 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:08.224941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:08.224964 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:10:08.225032 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:08.225040 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:08.225166 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:08.225285 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:08.227762 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.229430 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.231852 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.232814 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.232986 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.237273 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:08.237294 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:08.237320 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:08.237384 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:08.242357 (+ 4973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:08.243422 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:08.243434 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:08.243440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:08.243636 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:08.244202 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:08.244210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:08.244216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:08.244785 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:08.245115 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:08.246553 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0927 05:10:08.246826 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:10:08.248566 16969 tablet_replica.cc:330] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: stopping tablet replica
I20260927 05:10:08.250931 16660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:08.251151 16968 raft_consensus.cc:2229] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:10:08.252188 16968 raft_consensus.cc:2258] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:08.255648 16968 ts_tablet_manager.cc:1905] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:08.257735 16967 tablet_replica.cc:330] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: stopping tablet replica
I20260927 05:10:08.258232 16967 raft_consensus.cc:2229] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:10:08.258788 16967 raft_consensus.cc:2258] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:08.262004 16967 ts_tablet_manager.cc:1905] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:08.263527 16968 ts_tablet_manager.cc:1918] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:08.263723 16968 log.cc:1196] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/fb0b9c2bc2bd4b21bacd789082c49055
I20260927 05:10:08.264436 16968 ts_tablet_manager.cc:1939] T fb0b9c2bc2bd4b21bacd789082c49055 P 68ef1e853a264b39a83fc939de8ef703: Deleting consensus metadata
I20260927 05:10:08.266824 16466 catalog_manager.cc:4841] TS 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465): tablet fb0b9c2bc2bd4b21bacd789082c49055 (table test-table [id=a78e23abb3b54e21a368e514847dd71d]) successfully deleted
I20260927 05:10:08.269666 16967 ts_tablet_manager.cc:1918] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:08.269862 16967 log.cc:1196] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/fb0b9c2bc2bd4b21bacd789082c49055
I20260927 05:10:08.270437 16967 ts_tablet_manager.cc:1939] T fb0b9c2bc2bd4b21bacd789082c49055 P 9a928f8d66224576be6e2d8caf31ce04: Deleting consensus metadata
I20260927 05:10:08.272822 16464 catalog_manager.cc:4841] TS 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555): tablet fb0b9c2bc2bd4b21bacd789082c49055 (table test-table [id=a78e23abb3b54e21a368e514847dd71d]) successfully deleted
I20260927 05:10:08.250953 16969 raft_consensus.cc:2229] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:10:08.358708 16969 raft_consensus.cc:2258] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:08.358744 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 16516
I20260927 05:10:08.374485 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 16669
W20260927 05:10:08.374718 16463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33473 (error 108)
I20260927 05:10:08.377018 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.376464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:42800)
0927 05:10:08.376695 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.376826 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 05:10:08.377324 16463 catalog_manager.cc:4584] TS 00d4ba2781e54dc2955557054383c27d (127.0.97.1:33473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb0b9c2bc2bd4b21bacd789082c49055: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
I20260927 05:10:08.390282 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.375319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.0.97.2:44465)
0927 05:10:08.375483 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:08.375487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:08.375707 (+ 220us) server_negotiation.cc:407] Connection header received
0927 05:10:08.375970 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:08.375976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:08.376077 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:08.376193 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:08.377925 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.378706 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.380355 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.380776 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.380920 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.383640 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:08.383655 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:08.383672 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:08.383710 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:08.386972 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:08.387814 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:08.387827 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:08.387830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:08.388038 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:08.388471 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:08.388476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:08.388480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:08.388977 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:08.389298 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:08.389670 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 05:10:08.389912 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 05:10:08.390976 16792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:08.431114 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.430585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:42814)
0927 05:10:08.430847 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.430942 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 05:10:08.432255 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.431862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:42826)
0927 05:10:08.432070 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.432151 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 05:10:08.483582 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 16669
I20260927 05:10:08.493221 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.492612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:42828)
0927 05:10:08.492861 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.492996 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 05:10:08.494549 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.494068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:42842)
0927 05:10:08.494271 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.494372 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 05:10:08.502008 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 16800
I20260927 05:10:08.515668 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.502906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.0.97.3:38555)
0927 05:10:08.503070 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:08.503075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:08.503093 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:10:08.503153 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:08.503158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:08.503222 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:08.503339 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:08.504787 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.505534 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.506969 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:08.507372 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:08.507471 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:08.510223 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:08.510239 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:08.510255 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:08.510290 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:08.513077 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:08.513744 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:08.513752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:08.513754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:08.513893 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:08.514216 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:08.514220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:08.514222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:08.514563 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:08.514786 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:08.515076 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 05:10:08.515312 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 05:10:08.516510 16923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:08.575435 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.574907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:42856)
0927 05:10:08.575161 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.575263 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 05:10:08.576754 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:08.576299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33473 (local address 127.0.0.1:42862)
0927 05:10:08.576504 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:08.576597 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 05:10:08.607518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 16800
I20260927 05:10:08.624748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 16446
I20260927 05:10:08.625998 16509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:08.744613 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 16446
I20260927 05:10:28.770198 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34651
--webserver_interface=127.0.97.62
--webserver_port=36865
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:29.145004 16975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:29.145360 16975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:29.145457 16975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:29.153651 16975 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 05:10:29.153784 16975 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 05:10:29.153824 16975 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 05:10:29.153882 16975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:10:29.153940 16975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:29.154004 16975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:29.154053 16975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:10:29.154102 16975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:10:29.165300 16975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:29.166760 16975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:29.168428 16975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:29.182375 16980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:29.182276 16983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:29.182348 16981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:29.182804 16975 server_base.cc:1034] running on GCE node
I20260927 05:10:29.184195 16975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:29.186064 16975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:29.187433 16975 hybrid_clock.cc:648] HybridClock initialized: now 1790485829187313 us; error 50 us; skew 500 ppm
I20260927 05:10:29.187879 16975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:29 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:29.195288 16975 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:10:29.197245 16975 webserver.cc:460] Webserver started at http://127.0.97.62:36865/ using document root <none> and password file <none>
I20260927 05:10:29.198115 16975 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:29.198271 16975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:29.205049 16975 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 05:10:29.208611 16990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:29.210366 16975 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 05:10:29.210616 16975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "5587b12ee129401e83435f9cd8c84429"
format_stamp: "Formatted at 2026-09-27 05:09:43 on dist-test-slave-hgs0"
server_key: "6f911aebbc646d1dd45dd6e967cb51dc"
server_key_iv: "3e315f95f7d72a853bc237cb19e9ac6d"
server_key_version: "encryptionkey@0"
I20260927 05:10:29.211799 16975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:29.239868 16975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:29.245012 16975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:29.245354 16975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:29.267764 16975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34651
I20260927 05:10:29.267828 17041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34651 every 8 connection(s)
I20260927 05:10:29.269605 16975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:10:29.277750 17042 sys_catalog.cc:263] Verifying existing consensus state
I20260927 05:10:29.279243 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 16975
I20260927 05:10:29.280328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33473
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34651
--webserver_port=45937
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:10:29.281715 17042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Bootstrap starting.
I20260927 05:10:29.306015 17042 log.cc:824] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:29.326064 17042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:10:29.326746 17042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Bootstrap complete.
I20260927 05:10:29.338068 17042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:10:29.339588 17042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5587b12ee129401e83435f9cd8c84429, State: Initialized, Role: FOLLOWER
I20260927 05:10:29.340481 17042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:10:29.340785 17042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:10:29.340936 17042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:10:29.341161 17042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 2 FOLLOWER]: Advancing to term 3
I20260927 05:10:29.345752 17042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:10:29.346372 17042 leader_election.cc:302] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5587b12ee129401e83435f9cd8c84429; no voters:
I20260927 05:10:29.347155 17042 leader_election.cc:288] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 05:10:29.347311 17046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 05:10:29.348729 17046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [term 3 LEADER]: Becoming Leader. State: Replica: 5587b12ee129401e83435f9cd8c84429, State: Running, Role: LEADER
I20260927 05:10:29.349505 17046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } }
I20260927 05:10:29.349810 17042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:10:29.352115 17048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5587b12ee129401e83435f9cd8c84429. Latest consensus state: current_term: 3 leader_uuid: "5587b12ee129401e83435f9cd8c84429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } } }
I20260927 05:10:29.352598 17048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:29.352851 17047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5587b12ee129401e83435f9cd8c84429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5587b12ee129401e83435f9cd8c84429" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34651 } } }
I20260927 05:10:29.353298 17047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429 [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:29.364935 17055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:10:29.375265 17055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:10:29.377573 17055 catalog_manager.cc:1259] Loaded cluster ID: 123b2d849b7c4a0eb9741b94d6fc19dc
I20260927 05:10:29.377754 17055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:10:29.383234 17055 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:10:29.384836 17055 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Loaded TSK: 0
I20260927 05:10:29.385030 17055 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5587b12ee129401e83435f9cd8c84429: Loaded TSK: 1
F20260927 05:10:29.385957 17055 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f303ca8f9ad google::LogMessage::Fail() at ??:0
@ 0x7f303ca9288a google::LogMessage::SendToLog() at ??:0
@ 0x7f303ca8f4de google::LogMessage::Flush() at ??:0
@ 0x7f303ca90a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f30407f37d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f30407e273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f30524be1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f30524bd783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f30525492a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f30525490c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f30525935e9 std::function<>::operator()() at ??:0
@ 0x7f30524c1cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f30524bfc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3052547610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3052547439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f303f6db345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f303f6de91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f303f6de799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f303f6b9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f30429936db start_thread at ??:0
@ 0x7f303b92961f clone at ??:0
W20260927 05:10:29.712121 17044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:29.712504 17044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:29.712688 17044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:29.720525 17044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:29.720695 17044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:29.720896 17044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:10:29.731382 17044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45937
--tserver_master_addrs=127.0.97.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:29.732947 17044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:29.734568 17044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:29.750320 17090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:29.750311 17092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:29.750644 17089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:29.753825 17044 server_base.cc:1034] running on GCE node
I20260927 05:10:29.754608 17044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:29.755710 17044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:29.756961 17044 hybrid_clock.cc:648] HybridClock initialized: now 1790485829756930 us; error 77 us; skew 500 ppm
I20260927 05:10:29.757365 17044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:29 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:29.760952 17044 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:10:29.762672 17044 webserver.cc:460] Webserver started at http://127.0.97.1:45937/ using document root <none> and password file <none>
I20260927 05:10:29.763453 17044 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:29.763592 17044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:29.769665 17044 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 05:10:29.773144 17099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:29.774887 17044 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 05:10:29.775128 17044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "00d4ba2781e54dc2955557054383c27d"
format_stamp: "Formatted at 2026-09-27 05:09:43 on dist-test-slave-hgs0"
server_key: "d39a9ff92fe0505df159b031081810e2"
server_key_iv: "e68cd55d198bba07f0856f2caa3eb951"
server_key_version: "encryptionkey@0"
I20260927 05:10:29.776350 17044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:29.794384 17044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:29.799484 17044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:29.799860 17044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:29.801098 17044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:29.803257 17106 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 05:10:29.809885 17044 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 05:10:29.810132 17044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260927 05:10:29.810364 17044 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 05:10:29.813984 17044 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 05:10:29.814085 17044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260927 05:10:29.814584 17106 tablet_bootstrap.cc:492] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Bootstrap starting.
I20260927 05:10:29.844977 17106 log.cc:824] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:29.859726 17044 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33473
I20260927 05:10:29.859754 17213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33473 every 8 connection(s)
I20260927 05:10:29.861292 17044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:10:29.861486 17106 tablet_bootstrap.cc:492] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 05:10:29.862057 17106 tablet_bootstrap.cc:492] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Bootstrap complete.
I20260927 05:10:29.863634 17106 ts_tablet_manager.cc:1397] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Time spent bootstrapping tablet: real 0.049s user 0.025s sys 0.019s
I20260927 05:10:29.867153 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:29.864698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:47597)
0927 05:10:29.865702 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:29.866019 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":455,"threads_started":1}
W20260927 05:10:29.867901 17214 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34651 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34651: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
I20260927 05:10:29.868942 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:29.868522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:38319)
0927 05:10:29.868703 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:29.868818 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 05:10:29.869547 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 17044
I20260927 05:10:29.869930 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:29.869545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:57787)
0927 05:10:29.869697 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:29.869792 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 05:10:29.870155 17214 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:10:29.870343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44465
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34651
--webserver_port=46093
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:10:29.872956 17106 raft_consensus.cc:348] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:29.874610 17106 raft_consensus.cc:729] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d4ba2781e54dc2955557054383c27d, State: Initialized, Role: FOLLOWER
I20260927 05:10:29.875437 17106 consensus_queue.cc:260] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:29.877430 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:29.876992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:37771)
0927 05:10:29.877189 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:29.877282 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 05:10:29.877559 17106 ts_tablet_manager.cc:1428] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d: Time spent starting tablet: real 0.014s user 0.013s sys 0.003s
I20260927 05:10:29.878623 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:29.878231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:39827)
0927 05:10:29.878427 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:29.878496 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 05:10:30.263638 17218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:30.264107 17218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:30.264295 17218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:30.275533 17218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:30.275687 17218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:30.275918 17218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:10:30.286991 17218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46093
--tserver_master_addrs=127.0.97.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:30.288589 17218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:30.290109 17218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:30.305176 17224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:30.305236 17226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:30.305332 17223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:30.306577 17218 server_base.cc:1034] running on GCE node
I20260927 05:10:30.307294 17218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:30.308413 17218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:30.309643 17218 hybrid_clock.cc:648] HybridClock initialized: now 1790485830309581 us; error 53 us; skew 500 ppm
I20260927 05:10:30.310036 17218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:30 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:30.313694 17218 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:10:30.315662 17218 webserver.cc:460] Webserver started at http://127.0.97.2:46093/ using document root <none> and password file <none>
I20260927 05:10:30.316620 17218 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:30.316789 17218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:30.323470 17218 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 05:10:30.326819 17233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:30.328541 17218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:10:30.328752 17218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "68ef1e853a264b39a83fc939de8ef703"
format_stamp: "Formatted at 2026-09-27 05:09:44 on dist-test-slave-hgs0"
server_key: "207b750ad9af11bb902737205bed0edb"
server_key_iv: "e28d54eac57cb590c0bec82aceb7c4e2"
server_key_version: "encryptionkey@0"
I20260927 05:10:30.329838 17218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:30.344141 17218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:30.349186 17218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:30.349539 17218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:30.350683 17218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:30.352366 17218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:30.352490 17218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:30.352640 17218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:30.352741 17218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:30.389978 17218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44465
I20260927 05:10:30.390086 17345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44465 every 8 connection(s)
I20260927 05:10:30.391639 17218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:10:30.397009 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:30.394921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:38393)
0927 05:10:30.395736 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:30.396007 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":301,"threads_started":1}
W20260927 05:10:30.397840 17346 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34651 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34651: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
I20260927 05:10:30.398794 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:30.398433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:49109)
0927 05:10:30.398599 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:30.398676 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 05:10:30.399770 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:30.399370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:49683)
0927 05:10:30.399563 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:30.399647 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 05:10:30.400071 17346 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:10:30.400723 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 17218
I20260927 05:10:30.401356 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38555
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34651
--webserver_port=43289
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:30.773123 17349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:30.773464 17349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:30.773567 17349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:30.782392 17349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:30.782531 17349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:30.782658 17349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:10:30.793787 17349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43289
--tserver_master_addrs=127.0.97.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:30.795202 17349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:30.796612 17349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:30.812122 17354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:30.812091 17355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:30.812151 17357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:30.813603 17349 server_base.cc:1034] running on GCE node
I20260927 05:10:30.814365 17349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:30.815557 17349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:30.816810 17349 hybrid_clock.cc:648] HybridClock initialized: now 1790485830816726 us; error 84 us; skew 500 ppm
I20260927 05:10:30.817209 17349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:30 dist-test-slave-hgs0 krb5kdc[15892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:30.820726 17349 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:10:30.822535 17349 webserver.cc:460] Webserver started at http://127.0.97.3:43289/ using document root <none> and password file <none>
I20260927 05:10:30.823362 17349 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:30.823531 17349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:30.830265 17349 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 05:10:30.834295 17364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:30.836349 17349 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:10:30.836571 17349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "9a928f8d66224576be6e2d8caf31ce04"
format_stamp: "Formatted at 2026-09-27 05:09:45 on dist-test-slave-hgs0"
server_key: "d6649204c886593805eb58290581e2e3"
server_key_iv: "33c608f44d25cc47975a38a590c00c8d"
server_key_version: "encryptionkey@0"
I20260927 05:10:30.837651 17349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:30.875268 17349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:30.879901 17349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:30.880362 17349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:30.880645 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:30.879622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:54223)
0927 05:10:30.880265 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:30.880361 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":346,"threads_started":1}
I20260927 05:10:30.881671 17349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:30.883116 17349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:30.883248 17349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:30.883394 17349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:30.883486 17349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:30.919171 17349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38555
I20260927 05:10:30.919246 17477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38555 every 8 connection(s)
I20260927 05:10:30.921003 17349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:10:30.926723 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:30.924685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:40273)
0927 05:10:30.925500 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:30.925722 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":313,"threads_started":1}
W20260927 05:10:30.927482 17478 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34651 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34651: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
I20260927 05:10:30.928454 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:30.928109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:58339)
0927 05:10:30.928258 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:30.928334 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 05:10:30.929421 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:30.929058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:50939)
0927 05:10:30.929193 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:30.929282 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 05:10:30.929751 17478 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 05:10:30.929949 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 17349
I20260927 05:10:31.394006 17482 raft_consensus.cc:482] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:10:31.394372 17482 raft_consensus.cc:504] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:31.404604 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:31.400868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:33057)
0927 05:10:31.403997 (+ 3129us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:31.404158 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2910,"threads_started":1}
I20260927 05:10:31.411089 17482 leader_election.cc:288] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555), 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
Sep 27 05:10:31 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 27 05:10:31 dist-test-slave-hgs0 krb5kdc[15892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:10:31.439261 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:31.398908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38555 (local address 127.0.97.1:33323)
0927 05:10:31.406764 (+ 7856us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:31.406836 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:31.407662 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:31.413563 (+ 5901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:31.413579 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:31.414796 (+ 1217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:31.416116 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:31.416160 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:31.419395 (+ 3235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:31.419403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:31.421750 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:31.421770 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:31.423687 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:31.423694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:31.424094 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:31.425036 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:31.425067 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:31.428435 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:31.433587 (+ 5152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:31.433599 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:31.433622 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:31.434741 (+ 1119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:31.435452 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:31.435459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:31.435465 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:31.435728 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:31.437240 (+ 1512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:31.437251 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:31.437772 (+ 521us) client_negotiation.cc:769] Sending connection context
0927 05:10:31.438132 (+ 360us) client_negotiation.cc:240] Negotiation successful
0927 05:10:31.438761 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7680,"thread_start_us":9923,"threads_started":1}
I20260927 05:10:31.439373 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:31.400100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33323 (local address 127.0.97.3:38555)
0927 05:10:31.405667 (+ 5567us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:31.406070 (+ 403us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:31.407015 (+ 945us) server_negotiation.cc:407] Connection header received
0927 05:10:31.410931 (+ 3916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:31.410953 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:31.411728 (+ 775us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:31.412287 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:31.416472 (+ 4185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:31.419115 (+ 2643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:31.421976 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:31.423435 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:31.423678 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:31.428804 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:31.428813 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:31.428831 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:31.428878 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:31.433314 (+ 4436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:31.434980 (+ 1666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:31.434995 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:31.435008 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:31.435291 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:31.435902 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:31.435908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:31.435912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:31.436708 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:31.437308 (+ 600us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:31.438422 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0927 05:10:31.438890 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5338,"thread_start_us":5191,"threads_started":1}
W20260927 05:10:31.444862 17101 leader_election.cc:341] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555): Not found: Tablet not found: fb0b9c2bc2bd4b21bacd789082c49055
I20260927 05:10:31.445626 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:31.405758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44465 (local address 127.0.97.1:50437)
0927 05:10:31.417776 (+ 12018us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:31.417808 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:31.418001 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:31.422421 (+ 4420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:31.422426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:31.423141 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:31.423500 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:31.423511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:31.426512 (+ 3001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:31.426523 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:31.427695 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:31.427712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:31.429703 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:31.429714 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:31.429918 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:31.430510 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:31.430543 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:31.434033 (+ 3490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:31.441546 (+ 7513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:31.441553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:31.441557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:31.442238 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:31.442889 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:31.442898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:31.442902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:31.443175 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:31.444467 (+ 1292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:31.444471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:31.444629 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 05:10:31.444802 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 05:10:31.445174 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11813,"thread_start_us":12811,"threads_started":1}
I20260927 05:10:31.446118 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:31.406196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50437 (local address 127.0.97.2:44465)
0927 05:10:31.418689 (+ 12493us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:31.418706 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:31.418762 (+ 56us) server_negotiation.cc:407] Connection header received
0927 05:10:31.421384 (+ 2622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:31.421402 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:31.422072 (+ 670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:31.422475 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:31.423774 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:31.425838 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:31.427893 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:31.429453 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:31.429691 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:31.434451 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:31.434477 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:31.434496 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:31.434543 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:31.441286 (+ 6743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:31.442431 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:31.442446 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:31.442455 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:31.442657 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:31.443317 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:31.443324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:31.443334 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:31.444146 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:31.444365 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:31.445089 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 05:10:31.445668 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12286,"thread_start_us":12104,"threads_started":1}
W20260927 05:10:31.449805 17103 leader_election.cc:341] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465): Not found: Tablet not found: fb0b9c2bc2bd4b21bacd789082c49055
I20260927 05:10:31.450114 17103 leader_election.cc:302] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00d4ba2781e54dc2955557054383c27d; no voters: 68ef1e853a264b39a83fc939de8ef703, 9a928f8d66224576be6e2d8caf31ce04
I20260927 05:10:31.450680 17482 raft_consensus.cc:2726] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:10:31.882128 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:31.881656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:33657)
0927 05:10:31.881836 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:31.881941 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 05:10:31.931500 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:31.930506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:53813)
0927 05:10:31.931104 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:31.931239 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":305,"threads_started":1}
I20260927 05:10:32.407650 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:32.406929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:58397)
0927 05:10:32.407391 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:32.407478 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260927 05:10:32.884105 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:32.883147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:41611)
0927 05:10:32.883704 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:32.883801 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":275,"threads_started":1}
I20260927 05:10:32.933774 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:32.932855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:39251)
0927 05:10:32.933506 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:32.933606 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":317,"threads_started":1}
I20260927 05:10:33.146894 17492 raft_consensus.cc:482] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:10:33.147137 17492 raft_consensus.cc:504] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:33.147920 17492 leader_election.cc:288] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555), 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
W20260927 05:10:33.149119 17103 leader_election.cc:341] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465): Not found: Tablet not found: fb0b9c2bc2bd4b21bacd789082c49055
W20260927 05:10:33.149384 17101 leader_election.cc:341] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555): Not found: Tablet not found: fb0b9c2bc2bd4b21bacd789082c49055
I20260927 05:10:33.149561 17101 leader_election.cc:302] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00d4ba2781e54dc2955557054383c27d; no voters: 68ef1e853a264b39a83fc939de8ef703, 9a928f8d66224576be6e2d8caf31ce04
I20260927 05:10:33.149825 17492 raft_consensus.cc:2726] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:10:33.409500 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:33.408702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:36511)
0927 05:10:33.409168 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:33.409255 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":248,"threads_started":1}
I20260927 05:10:33.886073 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:33.885190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:54401)
0927 05:10:33.885809 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:33.885895 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":300,"threads_started":1}
I20260927 05:10:33.935563 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:33.934743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:50341)
0927 05:10:33.935303 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:33.935397 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":308,"threads_started":1}
I20260927 05:10:34.411513 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:34.410645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:35491)
0927 05:10:34.411235 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:34.411325 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":342,"threads_started":1}
I20260927 05:10:34.888386 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:34.887457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:57587)
0927 05:10:34.888066 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:34.888171 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":268,"threads_started":1}
I20260927 05:10:34.937290 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:34.936548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:42385)
0927 05:10:34.937010 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:34.937098 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260927 05:10:35.181150 17499 raft_consensus.cc:482] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:10:35.181397 17499 raft_consensus.cc:504] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:35.182205 17499 leader_election.cc:288] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555), 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
W20260927 05:10:35.183293 17101 leader_election.cc:341] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555): Not found: Tablet not found: fb0b9c2bc2bd4b21bacd789082c49055
W20260927 05:10:35.183465 17103 leader_election.cc:341] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465): Not found: Tablet not found: fb0b9c2bc2bd4b21bacd789082c49055
I20260927 05:10:35.183610 17103 leader_election.cc:302] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00d4ba2781e54dc2955557054383c27d; no voters: 68ef1e853a264b39a83fc939de8ef703, 9a928f8d66224576be6e2d8caf31ce04
I20260927 05:10:35.184006 17499 raft_consensus.cc:2726] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:10:35.413436 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:35.412568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:42821)
0927 05:10:35.413142 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:35.413258 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":281,"threads_started":1}
I20260927 05:10:35.890406 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:35.889567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:41115)
0927 05:10:35.890092 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:35.890213 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":284,"threads_started":1}
I20260927 05:10:35.939270 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:35.938493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:49443)
0927 05:10:35.938994 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:35.939095 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":261,"threads_started":1}
I20260927 05:10:36.415344 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:36.414524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:40465)
0927 05:10:36.415050 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:36.415142 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":283,"threads_started":1}
I20260927 05:10:36.892431 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:36.891623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:39261)
0927 05:10:36.892128 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:36.892214 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":240,"threads_started":1}
I20260927 05:10:36.941349 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:36.940429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:38043)
0927 05:10:36.940962 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:36.941107 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":270,"threads_started":1}
I20260927 05:10:37.417402 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:37.416522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:52063)
0927 05:10:37.417084 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:37.417182 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":257,"threads_started":1}
I20260927 05:10:37.894302 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:37.893405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:53521)
0927 05:10:37.893954 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:37.894098 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
I20260927 05:10:37.943207 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:37.942405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:37943)
0927 05:10:37.942927 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:37.943015 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":205,"threads_started":1}
I20260927 05:10:38.419096 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:38.418397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:36939)
0927 05:10:38.418825 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:38.418925 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260927 05:10:38.896515 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:38.895617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:41515)
0927 05:10:38.896161 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:38.896259 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":251,"threads_started":1}
I20260927 05:10:38.944983 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:38.944235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:38437)
0927 05:10:38.944688 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:38.944778 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":235,"threads_started":1}
I20260927 05:10:39.420806 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:39.420094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:48667)
0927 05:10:39.420549 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:39.420631 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":216,"threads_started":1}
I20260927 05:10:39.863018 17513 raft_consensus.cc:482] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:10:39.863260 17513 raft_consensus.cc:504] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:39.863965 17513 leader_election.cc:288] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555), 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
W20260927 05:10:39.864956 17101 leader_election.cc:341] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555): Not found: Tablet not found: fb0b9c2bc2bd4b21bacd789082c49055
W20260927 05:10:39.865104 17103 leader_election.cc:341] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465): Not found: Tablet not found: fb0b9c2bc2bd4b21bacd789082c49055
I20260927 05:10:39.865285 17103 leader_election.cc:302] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00d4ba2781e54dc2955557054383c27d; no voters: 68ef1e853a264b39a83fc939de8ef703, 9a928f8d66224576be6e2d8caf31ce04
I20260927 05:10:39.865559 17513 raft_consensus.cc:2726] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:10:39.898494 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:39.897622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:37991)
0927 05:10:39.898141 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:39.898281 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":275,"threads_started":1}
I20260927 05:10:39.946672 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:39.945927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:60035)
0927 05:10:39.946434 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:39.946517 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":235,"threads_started":1}
I20260927 05:10:40.422804 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:40.421836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:41113)
0927 05:10:40.422457 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:40.422571 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":324,"threads_started":1}
I20260927 05:10:40.900478 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:40.899571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:58939)
0927 05:10:40.900159 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:40.900247 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":240,"threads_started":1}
I20260927 05:10:40.948460 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:40.947666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:60633)
0927 05:10:40.948186 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:40.948272 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":301,"threads_started":1}
I20260927 05:10:41.429029 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:41.428197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:41399)
0927 05:10:41.428746 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:41.428845 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":252,"threads_started":1}
I20260927 05:10:41.902628 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:41.901679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:34941)
0927 05:10:41.902374 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:41.902460 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":424,"threads_started":1}
I20260927 05:10:41.950044 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:41.949321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:42129)
0927 05:10:41.949776 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:41.949868 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":222,"threads_started":1}
I20260927 05:10:42.430697 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:42.429971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:38953)
0927 05:10:42.430430 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:42.430515 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":230,"threads_started":1}
I20260927 05:10:42.585459 17523 raft_consensus.cc:482] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:10:42.585693 17523 raft_consensus.cc:504] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a928f8d66224576be6e2d8caf31ce04" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38555 } } peers { permanent_uuid: "00d4ba2781e54dc2955557054383c27d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33473 } } peers { permanent_uuid: "68ef1e853a264b39a83fc939de8ef703" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44465 } }
I20260927 05:10:42.586357 17523 leader_election.cc:288] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555), 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465)
W20260927 05:10:42.587311 17101 leader_election.cc:341] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9a928f8d66224576be6e2d8caf31ce04 (127.0.97.3:38555): Not found: Tablet not found: fb0b9c2bc2bd4b21bacd789082c49055
W20260927 05:10:42.587486 17103 leader_election.cc:341] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 68ef1e853a264b39a83fc939de8ef703 (127.0.97.2:44465): Not found: Tablet not found: fb0b9c2bc2bd4b21bacd789082c49055
I20260927 05:10:42.587599 17103 leader_election.cc:302] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00d4ba2781e54dc2955557054383c27d; no voters: 68ef1e853a264b39a83fc939de8ef703, 9a928f8d66224576be6e2d8caf31ce04
I20260927 05:10:42.587829 17523 raft_consensus.cc:2726] T fb0b9c2bc2bd4b21bacd789082c49055 P 00d4ba2781e54dc2955557054383c27d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 05:10:42.904799 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:42.903687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:43973)
0927 05:10:42.904425 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:42.904531 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":491,"threads_started":1}
I20260927 05:10:42.951826 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:42.951068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:35305)
0927 05:10:42.951562 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:42.951652 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":276,"threads_started":1}
I20260927 05:10:43.432365 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:43.431631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:39899)
0927 05:10:43.432073 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:43.432150 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":209,"threads_started":1}
I20260927 05:10:43.906667 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:43.905914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:44947)
0927 05:10:43.906397 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:43.906484 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":254,"threads_started":1}
I20260927 05:10:43.953631 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:43.952810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:55159)
0927 05:10:43.953340 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:43.953435 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":240,"threads_started":1}
I20260927 05:10:44.434374 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:44.433348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:49449)
0927 05:10:44.433940 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:44.434083 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":290,"threads_started":1}
I20260927 05:10:44.908663 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:44.907894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:36607)
0927 05:10:44.908410 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:44.908497 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":234,"threads_started":1}
I20260927 05:10:44.955469 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:44.954569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:44873)
0927 05:10:44.955101 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:44.955211 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":302,"threads_started":1}
I20260927 05:10:45.436134 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:45.435396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.2:37989)
0927 05:10:45.435838 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:45.435926 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
I20260927 05:10:45.910415 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:45.909601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.1:52441)
0927 05:10:45.910107 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:45.910220 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":253,"threads_started":1}
I20260927 05:10:45.936448 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 17044
I20260927 05:10:45.957397 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:45.956619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34651 (local address 127.0.97.3:45799)
0927 05:10:45.957127 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:45.957219 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":270,"threads_started":1}
I20260927 05:10:45.960224 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:45.937600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.0.97.1:33473)
0927 05:10:45.938263 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:45.938273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:45.938597 (+ 324us) server_negotiation.cc:407] Connection header received
0927 05:10:45.938805 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:45.938812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:45.939079 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:45.939226 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:45.940626 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:45.941664 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:45.942966 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:45.943561 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:45.943672 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:45.954047 (+ 10375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:45.954061 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:45.954077 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:45.954123 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:45.957380 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:45.958253 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:45.958270 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:45.958273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:45.958402 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:45.958737 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:45.958741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:45.958742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:45.959094 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:45.959351 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:45.959618 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 05:10:45.959815 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":314,"threads_started":1}
I20260927 05:10:45.961587 17209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:46.053151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 17044
I20260927 05:10:46.070566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 17218
I20260927 05:10:46.083823 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:46.071394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.0.97.2:44465)
0927 05:10:46.071892 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:46.071896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:46.071911 (+ 15us) server_negotiation.cc:407] Connection header received
0927 05:10:46.072009 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:46.072017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:46.072077 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:46.072192 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:46.073475 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:46.074276 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:46.075461 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:46.075888 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:46.076131 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:46.078632 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:46.078637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:46.078641 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:46.078672 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:46.081232 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:46.081920 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:46.081927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:46.081930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:46.082057 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:46.082417 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:46.082424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:46.082427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:46.082686 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:46.082946 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:46.083236 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 05:10:46.083454 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":269,"threads_started":1}
I20260927 05:10:46.084614 17341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:46.169835 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 17218
I20260927 05:10:46.187300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 17349
I20260927 05:10:46.201607 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:46.188158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.0.97.3:38555)
0927 05:10:46.188746 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:46.188751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:46.188770 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:10:46.188825 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:46.188828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:46.188887 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:46.188985 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:46.190451 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:46.191246 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:46.192621 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:46.193066 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:46.193174 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:46.195693 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:46.195699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:46.195703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:46.195738 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:46.199109 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:46.199739 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:46.199743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:46.199746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:46.199870 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:46.200277 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:46.200281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:46.200284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:46.200582 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:46.200800 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:46.201017 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 05:10:46.201187 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":302,"threads_started":1}
I20260927 05:10:46.202354 17473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:46.285908 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 17349
2026-09-27T05:10:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64854 ms)
[----------] 30 tests from SecurityITest (310864 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:10:46 dist-test-slave-hgs0 krb5kdc[17543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:10:46 dist-test-slave-hgs0 krb5kdc[17543](info): set up 2 sockets
Sep 27 05:10:46 dist-test-slave-hgs0 krb5kdc[17543](info): commencing operation
krb5kdc: starting...
W20260927 05:10:47.408466 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:10:47 dist-test-slave-hgs0 krb5kdc[17543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:10:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:10:47.656929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37711
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:47.984663 17556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:47.985005 17556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:47.985229 17556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:47.992488 17556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:10:47.992630 17556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:47.992733 17556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:47.992832 17556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:10:47.992903 17556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:10:48.002836 17556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:48.004244 17556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:48.005551 17556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:48.017306 17562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:48.017473 17561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:48.017473 17564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:48.017781 17556 server_base.cc:1034] running on GCE node
I20260927 05:10:48.018853 17556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:48.020264 17556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:48.021497 17556 hybrid_clock.cc:648] HybridClock initialized: now 1790485848021448 us; error 36 us; skew 500 ppm
I20260927 05:10:48.021865 17556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:48 dist-test-slave-hgs0 krb5kdc[17543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:48.024935 17556 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:10:48.026467 17556 webserver.cc:460] Webserver started at http://127.0.97.62:42571/ using document root <none> and password file <none>
I20260927 05:10:48.027207 17556 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:48.027343 17556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:48.027658 17556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:48.030618 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "fddd084151d046008831fd841119c435"
format_stamp: "Formatted at 2026-09-27 05:10:48 on dist-test-slave-hgs0"
server_key: "02a26304cb0f6fc91c444861e8e37c9d"
server_key_iv: "8c5ebc6b37df6ec93d0f40482fbc7b3b"
server_key_version: "encryptionkey@0"
I20260927 05:10:48.031363 17556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "fddd084151d046008831fd841119c435"
format_stamp: "Formatted at 2026-09-27 05:10:48 on dist-test-slave-hgs0"
server_key: "02a26304cb0f6fc91c444861e8e37c9d"
server_key_iv: "8c5ebc6b37df6ec93d0f40482fbc7b3b"
server_key_version: "encryptionkey@0"
I20260927 05:10:48.036413 17556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 05:10:48.039722 17571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:48.040998 17556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:10:48.041172 17556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "fddd084151d046008831fd841119c435"
format_stamp: "Formatted at 2026-09-27 05:10:48 on dist-test-slave-hgs0"
server_key: "02a26304cb0f6fc91c444861e8e37c9d"
server_key_iv: "8c5ebc6b37df6ec93d0f40482fbc7b3b"
server_key_version: "encryptionkey@0"
I20260927 05:10:48.041358 17556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:48.061456 17556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:48.065771 17556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:48.066004 17556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:48.084404 17556 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37711
I20260927 05:10:48.084438 17622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37711 every 8 connection(s)
I20260927 05:10:48.085764 17556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:10:48.089514 17623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:48.089885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 17556
I20260927 05:10:48.090090 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:10:48.090737 388 external_mini_cluster.cc:1428] Setting key 2888492ee12545e3366e624bc2c956b7
I20260927 05:10:48.102448 17623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435: Bootstrap starting.
Sep 27 05:10:48 dist-test-slave-hgs0 krb5kdc[17543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:48.106583 17623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:48.107630 17623 log.cc:824] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:48.110582 17623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435: No bootstrap required, opened a new log
I20260927 05:10:48.115868 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:48.092780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.0.97.62:37711)
0927 05:10:48.093530 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:48.093542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:48.093582 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:10:48.095657 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:48.095682 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:48.096172 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:48.096659 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:48.098104 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:48.099399 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:48.100925 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:48.102047 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:48.102242 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:48.107418 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:48.107427 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:48.107442 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:48.107490 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:48.111050 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:48.111900 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:48.111909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:48.111915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:48.112194 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:48.112651 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:48.112660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:48.112664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:48.113333 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:48.113694 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:48.114241 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 05:10:48.114721 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":357,"threads_started":1}
I20260927 05:10:48.118744 17623 raft_consensus.cc:348] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd084151d046008831fd841119c435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37711 } }
I20260927 05:10:48.119197 17623 raft_consensus.cc:374] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:48.119313 17623 raft_consensus.cc:729] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddd084151d046008831fd841119c435, State: Initialized, Role: FOLLOWER
I20260927 05:10:48.119863 17623 consensus_queue.cc:260] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd084151d046008831fd841119c435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37711 } }
I20260927 05:10:48.120167 17623 raft_consensus.cc:388] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:10:48.120323 17623 raft_consensus.cc:482] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:10:48.120497 17623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:48.122665 17623 raft_consensus.cc:504] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd084151d046008831fd841119c435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37711 } }
I20260927 05:10:48.123183 17623 leader_election.cc:302] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fddd084151d046008831fd841119c435; no voters:
I20260927 05:10:48.123799 17623 leader_election.cc:288] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:10:48.123971 17628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:10:48.125147 17628 raft_consensus.cc:686] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [term 1 LEADER]: Becoming Leader. State: Replica: fddd084151d046008831fd841119c435, State: Running, Role: LEADER
I20260927 05:10:48.125779 17628 consensus_queue.cc:237] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd084151d046008831fd841119c435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37711 } }
I20260927 05:10:48.126097 17623 sys_catalog.cc:564] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:10:48.128551 17630 sys_catalog.cc:455] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fddd084151d046008831fd841119c435. Latest consensus state: current_term: 1 leader_uuid: "fddd084151d046008831fd841119c435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd084151d046008831fd841119c435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37711 } } }
I20260927 05:10:48.128933 17630 sys_catalog.cc:458] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:48.129456 17629 sys_catalog.cc:455] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fddd084151d046008831fd841119c435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fddd084151d046008831fd841119c435" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37711 } } }
I20260927 05:10:48.129776 17629 sys_catalog.cc:458] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435 [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:48.129945 17635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:10:48.136386 17635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:10:48.147702 17635 catalog_manager.cc:1347] Generated new cluster ID: 785778a2b584402b85201d8ddd845622
I20260927 05:10:48.147814 17635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:10:48.171669 17635 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:10:48.173051 17635 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:10:48.179652 17635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435: Generated new TSK 0
I20260927 05:10:48.180521 17635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:10:48.248482 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37711
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:48.577054 17649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:48.577338 17649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:48.577585 17649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:48.585175 17649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:48.585322 17649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:48.585489 17649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:10:48.595085 17649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:48.596374 17649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:48.597749 17649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:48.610880 17654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:48.610911 17655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:48.610939 17657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:48.611812 17649 server_base.cc:1034] running on GCE node
I20260927 05:10:48.612408 17649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:48.613392 17649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:48.614588 17649 hybrid_clock.cc:648] HybridClock initialized: now 1790485848614536 us; error 40 us; skew 500 ppm
I20260927 05:10:48.614898 17649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:48 dist-test-slave-hgs0 krb5kdc[17543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:48.617887 17649 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:10:48.619305 17649 webserver.cc:460] Webserver started at http://127.0.97.1:41389/ using document root <none> and password file <none>
I20260927 05:10:48.620090 17649 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:48.620215 17649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:48.620527 17649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:48.623404 17649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "692fd6ae28154aa28da233a14055b4c5"
format_stamp: "Formatted at 2026-09-27 05:10:48 on dist-test-slave-hgs0"
server_key: "3fd04bfc79e18533bbe01aae6f825090"
server_key_iv: "71e3af66ef9fcd3465a789b2b15710c0"
server_key_version: "encryptionkey@0"
I20260927 05:10:48.624245 17649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "692fd6ae28154aa28da233a14055b4c5"
format_stamp: "Formatted at 2026-09-27 05:10:48 on dist-test-slave-hgs0"
server_key: "3fd04bfc79e18533bbe01aae6f825090"
server_key_iv: "71e3af66ef9fcd3465a789b2b15710c0"
server_key_version: "encryptionkey@0"
I20260927 05:10:48.629451 17649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:10:48.632838 17664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:48.634107 17649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:10:48.634269 17649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "692fd6ae28154aa28da233a14055b4c5"
format_stamp: "Formatted at 2026-09-27 05:10:48 on dist-test-slave-hgs0"
server_key: "3fd04bfc79e18533bbe01aae6f825090"
server_key_iv: "71e3af66ef9fcd3465a789b2b15710c0"
server_key_version: "encryptionkey@0"
I20260927 05:10:48.634467 17649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:48.656165 17649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:48.660329 17649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:48.660616 17649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:48.661581 17649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:48.662932 17649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:48.663040 17649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:48.663182 17649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:48.663249 17649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:48.697162 17649 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41779
I20260927 05:10:48.697244 17776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41779 every 8 connection(s)
I20260927 05:10:48.698496 17649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:10:48.702874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 17649
I20260927 05:10:48.703083 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:10:48.703900 388 external_mini_cluster.cc:1428] Setting key 15fa61d653cbaf1991ca308445a87aba
Sep 27 05:10:48 dist-test-slave-hgs0 krb5kdc[17543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:48.720223 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:48.701030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41261 (local address 127.0.97.62:37711)
0927 05:10:48.701481 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:48.701485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:48.702338 (+ 853us) server_negotiation.cc:407] Connection header received
0927 05:10:48.705039 (+ 2701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:48.705044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:48.705127 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:48.705243 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:48.707140 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:48.707843 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:48.710274 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:48.710664 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:48.710755 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:48.714760 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:48.714765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:48.714769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:48.714801 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:48.717274 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:48.717975 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:48.717980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:48.717988 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:48.718102 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:48.718514 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:48.718518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:48.718522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:48.718785 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:48.718931 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:48.719617 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 05:10:48.719835 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":245,"threads_started":1}
I20260927 05:10:48.721156 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:48.701464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37711 (local address 127.0.97.1:41261)
0927 05:10:48.702184 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:48.702230 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:48.704727 (+ 2497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:48.705447 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:48.705454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:48.706153 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:48.706950 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:48.706981 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:48.708010 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:48.708018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:48.710133 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:48.710150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:48.710883 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:48.710892 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:48.711218 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:48.711988 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:48.712023 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:48.714547 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:48.717412 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:48.717421 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:48.717437 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:48.717853 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:48.718215 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:48.718220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:48.718224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:48.718400 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:48.718919 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:48.718926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:48.719251 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 05:10:48.719500 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 05:10:48.720048 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":282,"threads_started":1}
I20260927 05:10:48.722663 17777 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37711
I20260927 05:10:48.723002 17777 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:48.723912 17777 heartbeater.cc:511] Master 127.0.97.62:37711 requested a full tablet report, sending...
I20260927 05:10:48.726291 17588 ts_manager.cc:194] Registered new tserver with Master: 692fd6ae28154aa28da233a14055b4c5 (127.0.97.1:41779)
I20260927 05:10:48.727921 17588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41261
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:10:48.761425 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37711
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:49.090803 17783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:49.091130 17783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:49.091303 17783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:49.099188 17783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:49.099341 17783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:49.099509 17783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:10:49.111169 17783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:49.112655 17783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:49.113999 17783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:49.127326 17791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:49.127470 17788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:49.127426 17789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:49.128937 17783 server_base.cc:1034] running on GCE node
I20260927 05:10:49.129518 17783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:49.130452 17783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:49.131668 17783 hybrid_clock.cc:648] HybridClock initialized: now 1790485849131584 us; error 60 us; skew 500 ppm
I20260927 05:10:49.132002 17783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:49 dist-test-slave-hgs0 krb5kdc[17543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:49.135345 17783 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:10:49.136936 17783 webserver.cc:460] Webserver started at http://127.0.97.2:37747/ using document root <none> and password file <none>
I20260927 05:10:49.137708 17783 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:49.137836 17783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:49.138163 17783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:49.141286 17783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "71074cb7972d44fe9c49b4b060eba8e3"
format_stamp: "Formatted at 2026-09-27 05:10:49 on dist-test-slave-hgs0"
server_key: "93740dd36d8ef1aa5f1ef75e81eba5cb"
server_key_iv: "cf19936d9a5177a10bb53741e4669958"
server_key_version: "encryptionkey@0"
I20260927 05:10:49.142102 17783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "71074cb7972d44fe9c49b4b060eba8e3"
format_stamp: "Formatted at 2026-09-27 05:10:49 on dist-test-slave-hgs0"
server_key: "93740dd36d8ef1aa5f1ef75e81eba5cb"
server_key_iv: "cf19936d9a5177a10bb53741e4669958"
server_key_version: "encryptionkey@0"
I20260927 05:10:49.147679 17783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:10:49.151480 17798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:49.153081 17783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:10:49.153270 17783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "71074cb7972d44fe9c49b4b060eba8e3"
format_stamp: "Formatted at 2026-09-27 05:10:49 on dist-test-slave-hgs0"
server_key: "93740dd36d8ef1aa5f1ef75e81eba5cb"
server_key_iv: "cf19936d9a5177a10bb53741e4669958"
server_key_version: "encryptionkey@0"
I20260927 05:10:49.153467 17783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:49.170892 17783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:49.175221 17783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:49.175527 17783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:49.176581 17783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:49.177923 17783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:49.177994 17783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:49.178151 17783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:49.178201 17783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:49.211611 17783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41985
I20260927 05:10:49.211647 17910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41985 every 8 connection(s)
I20260927 05:10:49.212961 17783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:10:49.214644 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 17783
I20260927 05:10:49.214828 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:10:49.215421 388 external_mini_cluster.cc:1428] Setting key b95e27f947a4db807534dd74abc18fe1
Sep 27 05:10:49 dist-test-slave-hgs0 krb5kdc[17543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:49.236903 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.215746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41307 (local address 127.0.97.62:37711)
0927 05:10:49.215915 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:49.215919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:49.217307 (+ 1388us) server_negotiation.cc:407] Connection header received
0927 05:10:49.220304 (+ 2997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:49.220308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:49.220378 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:49.220467 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:49.222621 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.223324 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.225882 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.226409 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.226629 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.231129 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:49.231135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:49.231138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:49.231170 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:49.233727 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:49.234470 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:49.234474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:49.234477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:49.234592 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:49.235043 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:49.235047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:49.235050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:49.235385 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:49.235535 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:49.236311 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 05:10:49.236542 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:10:49.238039 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.216362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37711 (local address 127.0.97.2:41307)
0927 05:10:49.217149 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:49.217191 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:49.219967 (+ 2776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:49.220743 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:49.220758 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:49.221562 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:49.222419 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.222447 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.223488 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.223497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.225707 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.225733 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.227013 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.227020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.227250 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.228115 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:49.228153 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:49.230878 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:49.233885 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:49.233894 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:49.233913 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:49.234334 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:49.234703 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:49.234708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:49.234712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:49.234906 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:49.235508 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:49.235516 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:49.235867 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 05:10:49.236193 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 05:10:49.236805 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":341,"threads_started":1}
I20260927 05:10:49.239640 17911 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37711
I20260927 05:10:49.239998 17911 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:49.240741 17911 heartbeater.cc:511] Master 127.0.97.62:37711 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:10:49.242233 17588 ts_manager.cc:194] Registered new tserver with Master: 71074cb7972d44fe9c49b4b060eba8e3 (127.0.97.2:41985)
I20260927 05:10:49.243053 17588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:10:49.275450 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37711
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:49.601902 17916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:49.602183 17916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:49.602279 17916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:49.609467 17916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:49.609579 17916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:49.609691 17916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:10:49.619261 17916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:49.620507 17916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:49.621737 17916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:49.635187 17921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:49.635244 17924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:49.635133 17922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:49.636099 17916 server_base.cc:1034] running on GCE node
I20260927 05:10:49.636744 17916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:49.637733 17916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:49.638989 17916 hybrid_clock.cc:648] HybridClock initialized: now 1790485849638901 us; error 79 us; skew 500 ppm
I20260927 05:10:49.639282 17916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:49 dist-test-slave-hgs0 krb5kdc[17543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:49.642459 17916 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:10:49.643975 17916 webserver.cc:460] Webserver started at http://127.0.97.3:46863/ using document root <none> and password file <none>
I20260927 05:10:49.644742 17916 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:49.644884 17916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:49.645215 17916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:49.648324 17916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "4ac1280465264553a94c6773c23b7a0d"
format_stamp: "Formatted at 2026-09-27 05:10:49 on dist-test-slave-hgs0"
server_key: "96027146034a22d11e4ee3a2428bca73"
server_key_iv: "4a9143876076dcef5ccd257385098418"
server_key_version: "encryptionkey@0"
I20260927 05:10:49.649156 17916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4ac1280465264553a94c6773c23b7a0d"
format_stamp: "Formatted at 2026-09-27 05:10:49 on dist-test-slave-hgs0"
server_key: "96027146034a22d11e4ee3a2428bca73"
server_key_iv: "4a9143876076dcef5ccd257385098418"
server_key_version: "encryptionkey@0"
I20260927 05:10:49.654217 17916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:10:49.657806 17931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:49.659250 17916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:10:49.659374 17916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "4ac1280465264553a94c6773c23b7a0d"
format_stamp: "Formatted at 2026-09-27 05:10:49 on dist-test-slave-hgs0"
server_key: "96027146034a22d11e4ee3a2428bca73"
server_key_iv: "4a9143876076dcef5ccd257385098418"
server_key_version: "encryptionkey@0"
I20260927 05:10:49.659536 17916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:49.686475 17916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:49.691277 17916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:49.691565 17916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:49.692618 17916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:49.694031 17916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:49.694128 17916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:49.694221 17916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:49.694270 17916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:49.729934 17916 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40071
I20260927 05:10:49.729979 18043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40071 every 8 connection(s)
I20260927 05:10:49.731127 17777 heartbeater.cc:503] Master 127.0.97.62:37711 was elected leader, sending a full tablet report...
I20260927 05:10:49.731525 17916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:10:49.739182 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 17916
I20260927 05:10:49.739392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:10:49.739908 388 external_mini_cluster.cc:1428] Setting key bc285b6c296008fb3464c98868a1e059
Sep 27 05:10:49 dist-test-slave-hgs0 krb5kdc[17543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:49.752123 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.734468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44039 (local address 127.0.97.62:37711)
0927 05:10:49.734639 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:49.734644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:49.735956 (+ 1312us) server_negotiation.cc:407] Connection header received
0927 05:10:49.738011 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:49.738016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:49.738104 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:49.738209 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:49.740203 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.741281 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.742906 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.743280 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.743373 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.746746 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:49.746751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:49.746754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:49.746786 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:49.749505 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:49.750192 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:49.750196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:49.750199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:49.750316 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:49.750661 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:49.750664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:49.750666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:49.750913 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:49.751019 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:49.751638 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 05:10:49.751803 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:10:49.753011 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.735040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37711 (local address 127.0.97.3:44039)
0927 05:10:49.735802 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:49.735839 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:49.737673 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:49.738446 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:49.738455 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:49.739163 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:49.740005 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.740032 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.741443 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.741449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.742741 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.742752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.743460 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.743464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.743658 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.744348 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:49.744369 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:49.746556 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:49.749631 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:49.749637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:49.749649 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:49.750047 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:49.750404 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:49.750407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:49.750409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:49.750565 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:49.751015 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:49.751021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:49.751300 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 05:10:49.751511 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 05:10:49.751881 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":279,"threads_started":1}
I20260927 05:10:49.754757 18044 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37711
I20260927 05:10:49.755074 18044 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:49.755839 18044 heartbeater.cc:511] Master 127.0.97.62:37711 requested a full tablet report, sending...
I20260927 05:10:49.757212 17588 ts_manager.cc:194] Registered new tserver with Master: 4ac1280465264553a94c6773c23b7a0d (127.0.97.3:40071)
I20260927 05:10:49.758106 17588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44039
I20260927 05:10:49.767637 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:10:49.784857 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.772089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.0.97.62:37711)
0927 05:10:49.772247 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:49.772252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:49.772774 (+ 522us) server_negotiation.cc:407] Connection header received
0927 05:10:49.772911 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:49.772915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:49.772973 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:49.773076 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:49.774401 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.775097 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.776405 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.776799 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.776890 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.779580 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:49.779584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:49.779588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:49.779618 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:49.782350 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:49.783056 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:49.783060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:49.783062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:49.783177 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:49.783530 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:49.783533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:49.783535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:49.783835 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:49.784170 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:49.784459 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 05:10:49.784623 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:10:49.788929 17588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:10:49.795678 17588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:10:49.828902 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.809017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.0.97.2:41985)
0927 05:10:49.819734 (+ 10717us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:49.819745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:49.820395 (+ 650us) server_negotiation.cc:407] Connection header received
0927 05:10:49.820504 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:49.820512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:49.820778 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:49.820957 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:49.821866 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.823341 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.826269 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.827712 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.827834 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.828025 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:49.828387 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 05:10:49.828525 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10576,"thread_start_us":10463,"threads_started":1}
I20260927 05:10:49.828965 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.807519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40071 (local address 127.0.0.1:38044)
0927 05:10:49.809091 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:49.809134 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:49.809292 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:49.820192 (+ 10900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:49.820200 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:49.820262 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:49.820720 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.820733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.823078 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.823084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.825793 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.825807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.827755 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.827762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.828038 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.828102 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 05:10:49.828244 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 05:10:49.828466 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"mutex_wait_us":44,"spinlock_wait_cycles":8704,"thread_start_us":1209,"threads_started":1}
I20260927 05:10:49.829814 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.808471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.0.97.3:40071)
0927 05:10:49.818588 (+ 10117us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:49.818602 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:49.818647 (+ 45us) server_negotiation.cc:407] Connection header received
0927 05:10:49.818777 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:49.818789 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:49.819127 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:49.820931 (+ 1804us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:49.821236 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.822938 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.825989 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.827555 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.827690 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.827976 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:49.828314 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 05:10:49.828482 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9910,"thread_start_us":9741,"threads_started":1}
I20260927 05:10:49.829864 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.807752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41985 (local address 127.0.0.1:47168)
0927 05:10:49.820127 (+ 12375us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:49.820147 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:49.820264 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:49.821194 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:49.821200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:49.821223 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:49.821651 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.821662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.823452 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.823457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.825797 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.825807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.828048 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.828053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.828221 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.828257 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:10:49.828326 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 05:10:49.829482 (+ 1156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12240,"mutex_wait_us":1071,"spinlock_wait_cycles":6784,"thread_start_us":12087,"threads_started":1}
I20260927 05:10:49.831332 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.809514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41779 (local address 127.0.0.1:51016)
0927 05:10:49.823077 (+ 13563us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:49.823094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:49.823248 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:49.824160 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:49.824164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:49.824189 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:49.824711 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.824725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.826360 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.826365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.828513 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.828524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.830669 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.830674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.830847 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.830877 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 05:10:49.830957 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 05:10:49.831077 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13395,"mutex_wait_us":487,"thread_start_us":13289,"threads_started":1}
I20260927 05:10:49.832077 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.809979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.0.97.1:41779)
0927 05:10:49.822876 (+ 12897us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:49.822884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:49.823225 (+ 341us) server_negotiation.cc:407] Connection header received
0927 05:10:49.823427 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:49.823436 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:49.823688 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:49.823897 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:49.824896 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.826070 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.828691 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.830213 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.830331 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.830545 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:49.831657 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0927 05:10:49.831774 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12732,"thread_start_us":12578,"threads_started":1}
I20260927 05:10:49.833863 17979 tablet_service.cc:1467] Processing CreateTablet for tablet 65f84166af334f2593603d9fc36651f7 (DEFAULT_TABLE table=test-table [id=db91a0d350204d0987957e214961887e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:49.833864 17846 tablet_service.cc:1467] Processing CreateTablet for tablet 65f84166af334f2593603d9fc36651f7 (DEFAULT_TABLE table=test-table [id=db91a0d350204d0987957e214961887e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:49.835341 17979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f84166af334f2593603d9fc36651f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:49.835391 17846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f84166af334f2593603d9fc36651f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:49.834951 17712 tablet_service.cc:1467] Processing CreateTablet for tablet 65f84166af334f2593603d9fc36651f7 (DEFAULT_TABLE table=test-table [id=db91a0d350204d0987957e214961887e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:49.836191 17712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f84166af334f2593603d9fc36651f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:49.847569 18062 tablet_bootstrap.cc:492] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: Bootstrap starting.
I20260927 05:10:49.849749 18063 tablet_bootstrap.cc:492] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: Bootstrap starting.
I20260927 05:10:49.851296 18062 tablet_bootstrap.cc:654] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:49.852625 18062 log.cc:824] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:49.853622 18063 tablet_bootstrap.cc:654] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:49.854245 18064 tablet_bootstrap.cc:492] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: Bootstrap starting.
I20260927 05:10:49.854853 18063 log.cc:824] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:49.855929 18062 tablet_bootstrap.cc:492] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: No bootstrap required, opened a new log
I20260927 05:10:49.856281 18062 ts_tablet_manager.cc:1397] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 05:10:49.857843 18063 tablet_bootstrap.cc:492] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: No bootstrap required, opened a new log
I20260927 05:10:49.858086 18063 ts_tablet_manager.cc:1397] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 05:10:49.858134 18064 tablet_bootstrap.cc:654] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:49.859208 18064 log.cc:824] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:49.861598 18064 tablet_bootstrap.cc:492] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: No bootstrap required, opened a new log
I20260927 05:10:49.861851 18064 ts_tablet_manager.cc:1397] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 05:10:49.867012 18062 raft_consensus.cc:348] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 } } peers { permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 } } peers { permanent_uuid: "692fd6ae28154aa28da233a14055b4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41779 } }
I20260927 05:10:49.867472 18062 raft_consensus.cc:374] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:49.867624 18062 raft_consensus.cc:729] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692fd6ae28154aa28da233a14055b4c5, State: Initialized, Role: FOLLOWER
I20260927 05:10:49.868304 18062 consensus_queue.cc:260] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 } } peers { permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 } } peers { permanent_uuid: "692fd6ae28154aa28da233a14055b4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41779 } }
I20260927 05:10:49.868492 18063 raft_consensus.cc:348] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 } } peers { permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 } } peers { permanent_uuid: "692fd6ae28154aa28da233a14055b4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41779 } }
I20260927 05:10:49.868963 18063 raft_consensus.cc:374] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:49.869202 18063 raft_consensus.cc:729] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71074cb7972d44fe9c49b4b060eba8e3, State: Initialized, Role: FOLLOWER
I20260927 05:10:49.869396 18064 raft_consensus.cc:348] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 } } peers { permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 } } peers { permanent_uuid: "692fd6ae28154aa28da233a14055b4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41779 } }
I20260927 05:10:49.870191 18064 raft_consensus.cc:374] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:49.870335 18064 raft_consensus.cc:729] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac1280465264553a94c6773c23b7a0d, State: Initialized, Role: FOLLOWER
I20260927 05:10:49.870198 18063 consensus_queue.cc:260] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 } } peers { permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 } } peers { permanent_uuid: "692fd6ae28154aa28da233a14055b4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41779 } }
I20260927 05:10:49.870942 18064 consensus_queue.cc:260] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 } } peers { permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 } } peers { permanent_uuid: "692fd6ae28154aa28da233a14055b4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41779 } }
I20260927 05:10:49.871503 17911 heartbeater.cc:503] Master 127.0.97.62:37711 was elected leader, sending a full tablet report...
I20260927 05:10:49.872110 18044 heartbeater.cc:503] Master 127.0.97.62:37711 was elected leader, sending a full tablet report...
I20260927 05:10:49.872624 18064 ts_tablet_manager.cc:1428] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: Time spent starting tablet: real 0.011s user 0.000s sys 0.011s
I20260927 05:10:49.874171 18062 ts_tablet_manager.cc:1428] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: Time spent starting tablet: real 0.018s user 0.015s sys 0.001s
I20260927 05:10:49.874670 18063 ts_tablet_manager.cc:1428] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260927 05:10:49.876646 18068 raft_consensus.cc:482] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:10:49.876948 18068 raft_consensus.cc:504] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 } } peers { permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 } } peers { permanent_uuid: "692fd6ae28154aa28da233a14055b4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41779 } }
I20260927 05:10:49.879014 18068 leader_election.cc:288] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ac1280465264553a94c6773c23b7a0d (127.0.97.3:40071), 71074cb7972d44fe9c49b4b060eba8e3 (127.0.97.2:41985)
I20260927 05:10:49.886277 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.879616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54549 (local address 127.0.97.3:40071)
0927 05:10:49.879818 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:49.879824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:49.879995 (+ 171us) server_negotiation.cc:407] Connection header received
0927 05:10:49.880289 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:49.880298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:49.880399 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:49.880518 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:49.880996 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.882479 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.884419 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.885316 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.885461 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.885566 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:49.885895 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 05:10:49.886035 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:10:49.886446 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.879259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40071 (local address 127.0.97.1:54549)
0927 05:10:49.879878 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:49.879900 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:49.880113 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:49.880540 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:49.880544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:49.880563 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:49.880864 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.880871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.882650 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.882654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.884256 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.884269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.885523 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.885529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.885716 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.885752 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 05:10:49.885848 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 05:10:49.885974 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":563,"threads_started":1}
I20260927 05:10:49.887300 17999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f84166af334f2593603d9fc36651f7" candidate_uuid: "692fd6ae28154aa28da233a14055b4c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac1280465264553a94c6773c23b7a0d" is_pre_election: true
I20260927 05:10:49.887777 17999 raft_consensus.cc:2454] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 692fd6ae28154aa28da233a14055b4c5 in term 0.
I20260927 05:10:49.888418 17666 leader_election.cc:302] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac1280465264553a94c6773c23b7a0d, 692fd6ae28154aa28da233a14055b4c5; no voters:
I20260927 05:10:49.888793 18068 raft_consensus.cc:2781] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:10:49.888952 18068 raft_consensus.cc:482] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:10:49.889062 18068 raft_consensus.cc:3037] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:49.890249 18072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.879366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41985 (local address 127.0.97.1:55315)
0927 05:10:49.884478 (+ 5112us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:49.884495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:49.884634 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:49.884949 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:49.884953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:49.884976 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:49.885322 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.885333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.886684 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.886690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.888122 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:49.888135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.889642 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.889647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:49.889777 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.889801 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 05:10:49.889891 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 05:10:49.889986 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4982,"thread_start_us":4828,"threads_started":1}
I20260927 05:10:49.890331 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.879781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55315 (local address 127.0.97.2:41985)
0927 05:10:49.879976 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:49.879982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:49.884583 (+ 4601us) server_negotiation.cc:407] Connection header received
0927 05:10:49.884766 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:49.884771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:49.884843 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:49.884935 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:49.885515 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.886533 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.888266 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.889404 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.889555 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.889663 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:49.889935 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 05:10:49.890066 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:10:49.891815 18068 raft_consensus.cc:504] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 } } peers { permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 } } peers { permanent_uuid: "692fd6ae28154aa28da233a14055b4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41779 } }
I20260927 05:10:49.892637 18068 leader_election.cc:288] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [CANDIDATE]: Term 1 election: Requested vote from peers 4ac1280465264553a94c6773c23b7a0d (127.0.97.3:40071), 71074cb7972d44fe9c49b4b060eba8e3 (127.0.97.2:41985)
I20260927 05:10:49.893095 17999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f84166af334f2593603d9fc36651f7" candidate_uuid: "692fd6ae28154aa28da233a14055b4c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac1280465264553a94c6773c23b7a0d"
I20260927 05:10:49.893139 17866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f84166af334f2593603d9fc36651f7" candidate_uuid: "692fd6ae28154aa28da233a14055b4c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71074cb7972d44fe9c49b4b060eba8e3" is_pre_election: true
I20260927 05:10:49.893324 17999 raft_consensus.cc:3037] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:49.893260 17865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65f84166af334f2593603d9fc36651f7" candidate_uuid: "692fd6ae28154aa28da233a14055b4c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71074cb7972d44fe9c49b4b060eba8e3"
I20260927 05:10:49.893612 17866 raft_consensus.cc:2454] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 692fd6ae28154aa28da233a14055b4c5 in term 0.
I20260927 05:10:49.895658 17999 raft_consensus.cc:2454] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692fd6ae28154aa28da233a14055b4c5 in term 1.
I20260927 05:10:49.896054 17666 leader_election.cc:302] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ac1280465264553a94c6773c23b7a0d, 692fd6ae28154aa28da233a14055b4c5; no voters: 71074cb7972d44fe9c49b4b060eba8e3
I20260927 05:10:49.896355 18068 raft_consensus.cc:2781] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:10:49.896816 18068 raft_consensus.cc:686] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 1 LEADER]: Becoming Leader. State: Replica: 692fd6ae28154aa28da233a14055b4c5, State: Running, Role: LEADER
I20260927 05:10:49.897279 18068 consensus_queue.cc:237] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 } } peers { permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 } } peers { permanent_uuid: "692fd6ae28154aa28da233a14055b4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41779 } }
I20260927 05:10:49.901366 17587 catalog_manager.cc:5495] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 692fd6ae28154aa28da233a14055b4c5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "692fd6ae28154aa28da233a14055b4c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692fd6ae28154aa28da233a14055b4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41779 } health_report { overall_health: HEALTHY } } }
I20260927 05:10:49.916852 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.911866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.0.97.1:41779)
0927 05:10:49.912062 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:49.912066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:49.912084 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:10:49.912311 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:49.912315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:49.912398 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:49.912516 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:10:49.912857 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.913547 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.914887 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.915274 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.915358 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.915808 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:10:49.916377 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:10:49.916483 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:49.916589 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 05:10:49.916655 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:10:49.924621 17866 raft_consensus.cc:3037] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:49.924871 17999 raft_consensus.cc:1261] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d [term 1 FOLLOWER]: Refusing update from remote peer 692fd6ae28154aa28da233a14055b4c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:10:49.925726 18073 consensus_queue.cc:1035] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ac1280465264553a94c6773c23b7a0d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:10:49.927549 17866 raft_consensus.cc:1261] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3 [term 1 FOLLOWER]: Refusing update from remote peer 692fd6ae28154aa28da233a14055b4c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:10:49.928437 18073 consensus_queue.cc:1035] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71074cb7972d44fe9c49b4b060eba8e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:10:49.936997 18077 mvcc.cc:205] Tried to move back new op lower bound from 7333830041282154496 to 7333830041182736384. Current Snapshot: MvccSnapshot[applied={T|T < 7333830041282154496}]
I20260927 05:10:49.953625 17586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:10:49.954012 17586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:10:49.958192 17586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fddd084151d046008831fd841119c435: Sending DeleteTablet for 3 replicas of tablet 65f84166af334f2593603d9fc36651f7
I20260927 05:10:49.959422 17846 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f84166af334f2593603d9fc36651f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47168
I20260927 05:10:49.959723 17712 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f84166af334f2593603d9fc36651f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51016
I20260927 05:10:49.959751 17979 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f84166af334f2593603d9fc36651f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38044
I20260927 05:10:49.960290 18085 tablet_replica.cc:330] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: stopping tablet replica
I20260927 05:10:49.960747 18085 raft_consensus.cc:2229] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:10:49.961241 18085 raft_consensus.cc:2258] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:49.961653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 17649
I20260927 05:10:49.963497 18085 ts_tablet_manager.cc:1905] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:49.973029 18085 ts_tablet_manager.cc:1918] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:49.973381 18085 log.cc:1196] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/65f84166af334f2593603d9fc36651f7
I20260927 05:10:49.974244 18085 ts_tablet_manager.cc:1939] T 65f84166af334f2593603d9fc36651f7 P 71074cb7972d44fe9c49b4b060eba8e3: Deleting consensus metadata
I20260927 05:10:49.976816 17572 catalog_manager.cc:4841] TS 71074cb7972d44fe9c49b4b060eba8e3 (127.0.97.2:41985): tablet 65f84166af334f2593603d9fc36651f7 (table test-table [id=db91a0d350204d0987957e214961887e]) successfully deleted
I20260927 05:10:49.977106 18086 tablet_replica.cc:330] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: stopping tablet replica
I20260927 05:10:49.977900 18086 raft_consensus.cc:2229] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:10:49.978542 18087 tablet_replica.cc:330] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: stopping tablet replica
I20260927 05:10:49.978669 18086 raft_consensus.cc:2258] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:49.979022 18087 raft_consensus.cc:2229] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:10:49.979561 18087 raft_consensus.cc:2258] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:49.981729 18087 ts_tablet_manager.cc:1905] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:49.987349 18087 ts_tablet_manager.cc:1918] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:49.987581 18087 log.cc:1196] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/65f84166af334f2593603d9fc36651f7
I20260927 05:10:49.988220 18087 ts_tablet_manager.cc:1939] T 65f84166af334f2593603d9fc36651f7 P 4ac1280465264553a94c6773c23b7a0d: Deleting consensus metadata
I20260927 05:10:49.989430 18086 ts_tablet_manager.cc:1905] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:49.990163 17573 catalog_manager.cc:4841] TS 4ac1280465264553a94c6773c23b7a0d (127.0.97.3:40071): tablet 65f84166af334f2593603d9fc36651f7 (table test-table [id=db91a0d350204d0987957e214961887e]) successfully deleted
Sep 27 05:10:49 dist-test-slave-hgs0 krb5kdc[17543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:10:49.995527 18086 ts_tablet_manager.cc:1918] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:49.995726 18086 log.cc:1196] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/65f84166af334f2593603d9fc36651f7
I20260927 05:10:49.996397 18086 ts_tablet_manager.cc:1939] T 65f84166af334f2593603d9fc36651f7 P 692fd6ae28154aa28da233a14055b4c5: Deleting consensus metadata
I20260927 05:10:49.997982 17573 catalog_manager.cc:4841] TS 692fd6ae28154aa28da233a14055b4c5 (127.0.97.1:41779): tablet 65f84166af334f2593603d9fc36651f7 (table test-table [id=db91a0d350204d0987957e214961887e]) successfully deleted
I20260927 05:10:50.001572 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:49.962941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.0.97.1:41779)
0927 05:10:49.963163 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:49.963169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:49.980690 (+ 17521us) server_negotiation.cc:407] Connection header received
0927 05:10:49.980902 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:49.980907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:49.980996 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:49.981119 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:49.982833 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.984078 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.986269 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:49.987012 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:49.987164 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:49.993640 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:49.993654 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:49.993677 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:49.993739 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:49.998537 (+ 4798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:49.999240 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:49.999250 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:49.999254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:49.999467 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:49.999850 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:49.999861 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:49.999865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:50.000349 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:50.000676 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:50.000868 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 05:10:50.001145 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 05:10:50.002487 17772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:50.106660 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 17649
I20260927 05:10:50.130206 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 17783
Sep 27 05:10:50 dist-test-slave-hgs0 krb5kdc[17543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:10:50.147989 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:50.131034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.0.97.2:41985)
0927 05:10:50.131202 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:50.131206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:50.131384 (+ 178us) server_negotiation.cc:407] Connection header received
0927 05:10:50.131579 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:50.131583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:50.131649 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:50.131736 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:50.133806 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:50.134974 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:50.136430 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:50.137099 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:50.137227 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:50.141952 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:50.141973 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:50.141998 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:50.142045 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:50.145531 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:50.146141 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:50.146149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:50.146151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:50.146279 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:50.146586 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:50.146589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:50.146591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:50.146956 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:50.147167 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:50.147419 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 05:10:50.147575 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:10:50.148711 17906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:50.238608 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 17783
I20260927 05:10:50.255522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 17916
Sep 27 05:10:50 dist-test-slave-hgs0 krb5kdc[17543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:10:50.271018 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:50.256337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.0.97.3:40071)
0927 05:10:50.256512 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:50.256518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:50.256540 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:10:50.256640 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:50.256647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:50.256742 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:50.256855 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:50.258196 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:50.258947 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:50.260332 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:50.260789 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:50.260923 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:50.265295 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:50.265313 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:50.265338 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:50.265374 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:50.268301 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:50.268972 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:50.268980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:50.268982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:50.269124 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:50.269497 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:50.269503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:50.269506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:50.269867 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:50.270158 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:50.270465 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 05:10:50.270692 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 05:10:50.271687 18039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:50.362138 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 17916
I20260927 05:10:50.378613 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 17556
I20260927 05:10:50.379874 17618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:50.490499 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 17556
2026-09-27T05:10:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4210 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T05:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:10:50Z Disabled control of system clock
I20260927 05:10:50.570278 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46753
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:10:50.915257 18096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:50.915614 18096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:50.915707 18096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:50.923450 18096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:10:50.923595 18096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:50.923641 18096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:50.923708 18096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:10:50.923760 18096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:10:50.933674 18096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:50.935002 18096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:50.937007 18096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:50.949786 18104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:50.949815 18101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:50.949849 18102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:50.950727 18096 server_base.cc:1034] running on GCE node
I20260927 05:10:50.951488 18096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:50.953048 18096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:50.954308 18096 hybrid_clock.cc:648] HybridClock initialized: now 1790485850954260 us; error 72 us; skew 500 ppm
I20260927 05:10:50.954675 18096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:10:50.962329 18096 webserver.cc:460] Webserver started at http://127.0.97.62:32901/ using document root <none> and password file <none>
I20260927 05:10:50.963280 18096 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:50.963433 18096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:50.963791 18096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:50.966987 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "fb31ae999fd841e5b37509e466e7274b"
format_stamp: "Formatted at 2026-09-27 05:10:50 on dist-test-slave-hgs0"
server_key: "6f21b9fbc3383c98d55c2175e0b71dae"
server_key_iv: "da1b301768bff8bdd921fa7879f75ffe"
server_key_version: "encryptionkey@0"
I20260927 05:10:50.967850 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "fb31ae999fd841e5b37509e466e7274b"
format_stamp: "Formatted at 2026-09-27 05:10:50 on dist-test-slave-hgs0"
server_key: "6f21b9fbc3383c98d55c2175e0b71dae"
server_key_iv: "da1b301768bff8bdd921fa7879f75ffe"
server_key_version: "encryptionkey@0"
I20260927 05:10:50.973338 18096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 05:10:50.976886 18110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:50.978314 18096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:10:50.978502 18096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "fb31ae999fd841e5b37509e466e7274b"
format_stamp: "Formatted at 2026-09-27 05:10:50 on dist-test-slave-hgs0"
server_key: "6f21b9fbc3383c98d55c2175e0b71dae"
server_key_iv: "da1b301768bff8bdd921fa7879f75ffe"
server_key_version: "encryptionkey@0"
I20260927 05:10:50.978736 18096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:51.000605 18096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:51.001477 18096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:51.001775 18096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:51.030508 18096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46753
I20260927 05:10:51.030530 18161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46753 every 8 connection(s)
I20260927 05:10:51.031886 18096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:10:51.035650 18162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:51.036603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 18096
I20260927 05:10:51.036796 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:10:51.037245 388 external_mini_cluster.cc:1428] Setting key 450b93d1e91216b2ff760b5fca9d3784
I20260927 05:10:51.049527 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b: Bootstrap starting.
I20260927 05:10:51.051033 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:51.039113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.0.97.62:46753)
0927 05:10:51.040190 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:51.040211 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:51.040256 (+ 45us) server_negotiation.cc:407] Connection header received
0927 05:10:51.041845 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:51.041865 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:51.042315 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:51.042636 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:51.043268 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:51.044559 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:51.046524 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:51.047229 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:51.047412 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:51.048678 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:51.048687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:51.048705 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:51.049256 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:51.049332 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:51.049353 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:51.049438 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:51.049701 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 05:10:51.050043 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":396,"threads_started":1}
I20260927 05:10:51.053238 18162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:51.054519 18162 log.cc:824] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:51.057142 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b: No bootstrap required, opened a new log
I20260927 05:10:51.065196 18162 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb31ae999fd841e5b37509e466e7274b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } }
I20260927 05:10:51.065552 18162 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:51.065631 18162 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb31ae999fd841e5b37509e466e7274b, State: Initialized, Role: FOLLOWER
I20260927 05:10:51.066169 18162 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb31ae999fd841e5b37509e466e7274b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } }
I20260927 05:10:51.066370 18162 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:10:51.066468 18162 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:10:51.066581 18162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:51.069123 18162 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb31ae999fd841e5b37509e466e7274b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } }
I20260927 05:10:51.069581 18162 leader_election.cc:302] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb31ae999fd841e5b37509e466e7274b; no voters:
I20260927 05:10:51.070134 18162 leader_election.cc:288] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:10:51.070295 18167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:10:51.071027 18167 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [term 1 LEADER]: Becoming Leader. State: Replica: fb31ae999fd841e5b37509e466e7274b, State: Running, Role: LEADER
I20260927 05:10:51.071650 18167 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb31ae999fd841e5b37509e466e7274b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } }
I20260927 05:10:51.072438 18162 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:10:51.074573 18169 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb31ae999fd841e5b37509e466e7274b. Latest consensus state: current_term: 1 leader_uuid: "fb31ae999fd841e5b37509e466e7274b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb31ae999fd841e5b37509e466e7274b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } } }
I20260927 05:10:51.075037 18169 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:51.075174 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb31ae999fd841e5b37509e466e7274b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb31ae999fd841e5b37509e466e7274b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46753 } } }
I20260927 05:10:51.076177 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:51.077446 18173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:10:51.082810 18173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:10:51.091243 18173 catalog_manager.cc:1347] Generated new cluster ID: b00244a808c94d3ab949d7bbbeb659f2
I20260927 05:10:51.091346 18173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:10:51.112545 18173 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:10:51.114395 18173 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:10:51.134886 18173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b: Generated new TSK 0
I20260927 05:10:51.135772 18173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:10:51.148919 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46753
--builtin_ntp_servers=127.0.97.20:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:10:51.487814 18186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:51.488215 18186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:51.488323 18186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:51.495446 18186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:51.495534 18186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:51.495635 18186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:10:51.504657 18186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:51.506043 18186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:51.507722 18186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:51.520579 18191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:51.520632 18192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:51.520673 18194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:51.521682 18186 server_base.cc:1034] running on GCE node
I20260927 05:10:51.522244 18186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:51.523221 18186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:51.524461 18186 hybrid_clock.cc:648] HybridClock initialized: now 1790485851524386 us; error 64 us; skew 500 ppm
I20260927 05:10:51.524775 18186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:10:51.527141 18186 webserver.cc:460] Webserver started at http://127.0.97.1:41957/ using document root <none> and password file <none>
I20260927 05:10:51.527791 18186 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:51.527876 18186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:51.528214 18186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:51.531047 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "704ac6b0db264c1fabbbdb04b15b8e7d"
format_stamp: "Formatted at 2026-09-27 05:10:51 on dist-test-slave-hgs0"
server_key: "0c58d97c394ae137b20fe27769b7c546"
server_key_iv: "1ac0cff0f648ba72e8201070cba67977"
server_key_version: "encryptionkey@0"
I20260927 05:10:51.531798 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "704ac6b0db264c1fabbbdb04b15b8e7d"
format_stamp: "Formatted at 2026-09-27 05:10:51 on dist-test-slave-hgs0"
server_key: "0c58d97c394ae137b20fe27769b7c546"
server_key_iv: "1ac0cff0f648ba72e8201070cba67977"
server_key_version: "encryptionkey@0"
I20260927 05:10:51.536891 18186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 05:10:51.540200 18200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:51.541484 18186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:10:51.541635 18186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "704ac6b0db264c1fabbbdb04b15b8e7d"
format_stamp: "Formatted at 2026-09-27 05:10:51 on dist-test-slave-hgs0"
server_key: "0c58d97c394ae137b20fe27769b7c546"
server_key_iv: "1ac0cff0f648ba72e8201070cba67977"
server_key_version: "encryptionkey@0"
I20260927 05:10:51.541836 18186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:51.557616 18186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:51.558318 18186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:51.558539 18186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:51.559371 18186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:51.560794 18186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:51.560877 18186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:51.560956 18186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:51.561010 18186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:51.593575 18186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35461
I20260927 05:10:51.593626 18312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35461 every 8 connection(s)
I20260927 05:10:51.594913 18186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:10:51.595764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 18186
I20260927 05:10:51.595928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:10:51.597435 388 external_mini_cluster.cc:1428] Setting key 2672f3561360cb1d9825c85d439def6c
I20260927 05:10:51.600502 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46753
--builtin_ntp_servers=127.0.97.20:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:10:51.608115 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:51.597988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57587 (local address 127.0.97.62:46753)
0927 05:10:51.598493 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:51.598497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:51.599872 (+ 1375us) server_negotiation.cc:407] Connection header received
0927 05:10:51.601763 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:51.601769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:51.601870 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:51.601985 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:51.603001 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:51.603717 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:51.605343 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:51.605751 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:51.605889 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:51.607248 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:51.607255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:51.607264 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:51.607288 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:51.607343 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:51.607362 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:51.607462 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:51.607675 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 05:10:51.607781 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
I20260927 05:10:51.609045 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:51.598633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46753 (local address 127.0.97.1:57587)
0927 05:10:51.599721 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:51.599761 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:51.601508 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:51.602213 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:51.602221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:51.602303 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:51.602841 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:51.602857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:51.603981 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:51.603986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:51.605197 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:51.605210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:51.605974 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:51.605983 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:51.606215 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:51.606922 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:10:51.606925 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:51.606962 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:10:51.607037 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:51.607465 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:51.607473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:51.607479 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:10:51.607630 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 05:10:51.607919 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":325,"threads_started":1}
I20260927 05:10:51.610553 18313 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46753
I20260927 05:10:51.610875 18313 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:51.611768 18313 heartbeater.cc:511] Master 127.0.97.62:46753 requested a full tablet report, sending...
I20260927 05:10:51.613868 18127 ts_manager.cc:194] Registered new tserver with Master: 704ac6b0db264c1fabbbdb04b15b8e7d (127.0.97.1:35461)
I20260927 05:10:51.615314 18127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:57587
W20260927 05:10:51.928834 18317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:51.929172 18317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:51.929338 18317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:51.936817 18317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:51.936959 18317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:51.937146 18317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:10:51.946635 18317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:51.947973 18317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:51.949800 18317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:51.963172 18322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:51.963194 18323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:51.963220 18325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:51.964076 18317 server_base.cc:1034] running on GCE node
I20260927 05:10:51.964756 18317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:51.965698 18317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:51.966957 18317 hybrid_clock.cc:648] HybridClock initialized: now 1790485851966902 us; error 68 us; skew 500 ppm
I20260927 05:10:51.967311 18317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:10:51.969662 18317 webserver.cc:460] Webserver started at http://127.0.97.2:39365/ using document root <none> and password file <none>
I20260927 05:10:51.970371 18317 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:51.970503 18317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:51.970841 18317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:51.973941 18317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "f55b48c02ea44f68b6834d44c47d8118"
format_stamp: "Formatted at 2026-09-27 05:10:51 on dist-test-slave-hgs0"
server_key: "e43b681405a52741b904759cb1190141"
server_key_iv: "02f415a93af0a28b775c9ad5223a9af8"
server_key_version: "encryptionkey@0"
I20260927 05:10:51.974730 18317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f55b48c02ea44f68b6834d44c47d8118"
format_stamp: "Formatted at 2026-09-27 05:10:51 on dist-test-slave-hgs0"
server_key: "e43b681405a52741b904759cb1190141"
server_key_iv: "02f415a93af0a28b775c9ad5223a9af8"
server_key_version: "encryptionkey@0"
I20260927 05:10:51.979926 18317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 05:10:51.983582 18331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:51.985134 18317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:10:51.985304 18317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "f55b48c02ea44f68b6834d44c47d8118"
format_stamp: "Formatted at 2026-09-27 05:10:51 on dist-test-slave-hgs0"
server_key: "e43b681405a52741b904759cb1190141"
server_key_iv: "02f415a93af0a28b775c9ad5223a9af8"
server_key_version: "encryptionkey@0"
I20260927 05:10:51.985522 18317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:52.005354 18317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:52.006219 18317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:52.006496 18317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:52.007434 18317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:52.008936 18317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:52.009060 18317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:52.009179 18317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:52.009269 18317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:52.043498 18317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38335
I20260927 05:10:52.043567 18443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38335 every 8 connection(s)
I20260927 05:10:52.044895 18317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:10:52.046290 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 18317
I20260927 05:10:52.046463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:10:52.046998 388 external_mini_cluster.cc:1428] Setting key ce11423e2f8f0d6b932e5fb69b332b6b
I20260927 05:10:52.050148 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46753
--builtin_ntp_servers=127.0.97.20:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:10:52.060891 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.049631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50609 (local address 127.0.97.62:46753)
0927 05:10:52.049845 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:52.049851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:52.051567 (+ 1716us) server_negotiation.cc:407] Connection header received
0927 05:10:52.053389 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:52.053397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:52.053487 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:52.053606 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:52.054642 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.055803 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.057846 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.058261 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.058362 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.060054 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:52.060065 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:52.060068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:52.060087 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:52.060117 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:52.060128 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:52.060225 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:52.060478 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 05:10:52.060577 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:10:52.061937 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.050293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46753 (local address 127.0.97.2:50609)
0927 05:10:52.051392 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:52.051436 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:52.053127 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:52.053795 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:52.053804 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:52.053893 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:52.054461 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.054478 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.056039 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.056046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.057688 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.057707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.058456 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.058461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.058809 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.059699 (+ 890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:10:52.059702 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:52.059742 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:10:52.059835 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:52.060222 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:52.060239 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:52.060251 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 05:10:52.060454 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 05:10:52.060845 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":366,"threads_started":1}
I20260927 05:10:52.063428 18444 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46753
I20260927 05:10:52.063751 18444 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:52.064633 18444 heartbeater.cc:511] Master 127.0.97.62:46753 requested a full tablet report, sending...
I20260927 05:10:52.066212 18127 ts_manager.cc:194] Registered new tserver with Master: f55b48c02ea44f68b6834d44c47d8118 (127.0.97.2:38335)
I20260927 05:10:52.067034 18127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:50609
W20260927 05:10:52.388756 18447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:52.389106 18447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:52.389240 18447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:52.396757 18447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:52.396925 18447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:52.397116 18447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:10:52.406487 18447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:52.407905 18447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:52.409795 18447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:52.423763 18452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:52.423719 18453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:52.423795 18455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:52.424854 18447 server_base.cc:1034] running on GCE node
I20260927 05:10:52.425467 18447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:52.426419 18447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:52.427630 18447 hybrid_clock.cc:648] HybridClock initialized: now 1790485852427562 us; error 58 us; skew 500 ppm
I20260927 05:10:52.427987 18447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:10:52.430603 18447 webserver.cc:460] Webserver started at http://127.0.97.3:43873/ using document root <none> and password file <none>
I20260927 05:10:52.431339 18447 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:52.431437 18447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:52.431746 18447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:52.435173 18447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "cd040dd0476046b3a4a8ebf3685c7281"
format_stamp: "Formatted at 2026-09-27 05:10:52 on dist-test-slave-hgs0"
server_key: "6383879e1b3643271a56bd4200783ef7"
server_key_iv: "6ab45926a7089560fd9b2d99d705b21f"
server_key_version: "encryptionkey@0"
I20260927 05:10:52.436177 18447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cd040dd0476046b3a4a8ebf3685c7281"
format_stamp: "Formatted at 2026-09-27 05:10:52 on dist-test-slave-hgs0"
server_key: "6383879e1b3643271a56bd4200783ef7"
server_key_iv: "6ab45926a7089560fd9b2d99d705b21f"
server_key_version: "encryptionkey@0"
I20260927 05:10:52.442230 18447 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:10:52.446134 18461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:52.447640 18447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:10:52.447844 18447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "cd040dd0476046b3a4a8ebf3685c7281"
format_stamp: "Formatted at 2026-09-27 05:10:52 on dist-test-slave-hgs0"
server_key: "6383879e1b3643271a56bd4200783ef7"
server_key_iv: "6ab45926a7089560fd9b2d99d705b21f"
server_key_version: "encryptionkey@0"
I20260927 05:10:52.448148 18447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:52.463706 18447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:52.464720 18447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:52.465039 18447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:52.466354 18447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:52.468256 18447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:52.468392 18447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:52.468549 18447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:52.468639 18447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:52.506193 18447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34291
I20260927 05:10:52.506253 18573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34291 every 8 connection(s)
I20260927 05:10:52.507633 18447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:10:52.508932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 18447
I20260927 05:10:52.509135 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:10:52.509665 388 external_mini_cluster.cc:1428] Setting key 49a9adb4311c690d307c97682a5214dd
I20260927 05:10:52.524426 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.510881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46161 (local address 127.0.97.62:46753)
0927 05:10:52.511136 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:52.511143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:52.513036 (+ 1893us) server_negotiation.cc:407] Connection header received
0927 05:10:52.515139 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:52.515144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:52.515257 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:52.515367 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:52.517062 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.518028 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.520268 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.521052 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.521213 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.523160 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:52.523167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:52.523171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:52.523197 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:52.523251 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:52.523266 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:52.523349 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:52.523702 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 05:10:52.523843 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 05:10:52.525872 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.511559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46753 (local address 127.0.97.3:46161)
0927 05:10:52.512831 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:52.512905 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:52.514810 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:52.515698 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:52.515710 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:52.515845 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:52.516784 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.516807 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.518299 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.518305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.520057 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.520075 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.521406 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.521413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.521712 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.522644 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:10:52.522647 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:52.522700 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:10:52.522798 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:52.523395 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:52.523410 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:52.523428 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 05:10:52.523620 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 05:10:52.524182 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":463,"threads_started":1}
I20260927 05:10:52.527423 18574 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46753
I20260927 05:10:52.527796 18574 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:52.528785 18574 heartbeater.cc:511] Master 127.0.97.62:46753 requested a full tablet report, sending...
I20260927 05:10:52.530441 18127 ts_manager.cc:194] Registered new tserver with Master: cd040dd0476046b3a4a8ebf3685c7281 (127.0.97.3:34291)
I20260927 05:10:52.531286 18127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46161
I20260927 05:10:52.539062 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:10:52.550616 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.543765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.0.97.62:46753)
0927 05:10:52.544177 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:52.544185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:52.544461 (+ 276us) server_negotiation.cc:407] Connection header received
0927 05:10:52.544717 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:52.544722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:52.544851 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:52.544986 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:52.545555 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.546383 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.547970 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.548412 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.548575 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.549833 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:52.549838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:52.549842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:52.549864 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:52.549901 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:52.549914 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:52.550023 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:52.550190 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 05:10:52.550300 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260927 05:10:52.555392 18127 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:10:52.563325 18127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:10:52.595377 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.579657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.0.97.1:35461)
0927 05:10:52.581580 (+ 1923us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:52.581595 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:52.581639 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:10:52.581824 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:52.581838 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:52.582181 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:52.582450 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:52.584067 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.586426 (+ 2359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.590947 (+ 4521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.593136 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.593305 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.593693 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:52.594741 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0927 05:10:52.594955 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1680,"thread_start_us":1528,"threads_started":1}
I20260927 05:10:52.596446 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.579315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35461 (local address 127.0.0.1:57546)
0927 05:10:52.580270 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:52.580330 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:52.580574 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:52.582836 (+ 2262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:52.582850 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:52.582891 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:52.583450 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.583471 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.586670 (+ 3199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.586682 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.590677 (+ 3995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.590697 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.593972 (+ 3275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.593982 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.594357 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.594437 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 05:10:52.595732 (+ 1295us) client_negotiation.cc:240] Negotiation successful
0927 05:10:52.595995 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":375,"threads_started":1}
I20260927 05:10:52.601717 18248 tablet_service.cc:1467] Processing CreateTablet for tablet d6e62a59145943f0a2ea7091f25323f9 (DEFAULT_TABLE table=test-table [id=e6d0a4a0324a4cf68ec6e2beb06624a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:52.603597 18248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e62a59145943f0a2ea7091f25323f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:52.605782 18587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.579607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38335 (local address 127.0.0.1:51802)
0927 05:10:52.590196 (+ 10589us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:52.590222 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:52.590362 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:52.591873 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:52.591881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:52.591919 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:52.592506 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.592523 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.599768 (+ 7245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.599774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.602431 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.602455 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.604818 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.604826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.605036 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.605075 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 05:10:52.605181 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 05:10:52.605346 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10369,"mutex_wait_us":101,"thread_start_us":10130,"threads_started":1}
I20260927 05:10:52.605845 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.580148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.0.97.2:38335)
0927 05:10:52.590925 (+ 10777us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:52.590936 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:52.590983 (+ 47us) server_negotiation.cc:407] Connection header received
0927 05:10:52.591128 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:52.591155 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:52.591556 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:52.597636 (+ 6080us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:52.597896 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.599555 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.602795 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.604539 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.604752 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.605038 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:52.605277 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 05:10:52.605465 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10585,"thread_start_us":10483,"threads_started":1}
I20260927 05:10:52.607867 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.591043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34291 (local address 127.0.0.1:43330)
0927 05:10:52.594830 (+ 3787us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:52.594852 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:52.594975 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:52.597174 (+ 2199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:52.597179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:52.597222 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:52.597582 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.597595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.599804 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.599811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.604580 (+ 4769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.604601 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.606913 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.606921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.607128 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.607190 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 05:10:52.607298 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 05:10:52.607457 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3647,"mutex_wait_us":87,"thread_start_us":4006,"threads_started":1}
I20260927 05:10:52.607916 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.590637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.0.97.3:34291)
0927 05:10:52.596156 (+ 5519us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:52.596166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:52.596214 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:10:52.596317 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:52.596327 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:52.596655 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:52.596876 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:52.597810 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.599620 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.604852 (+ 5232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.606629 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.606864 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.607209 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:52.607359 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 05:10:52.607530 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4944,"spinlock_wait_cycles":795904,"thread_start_us":4765,"threads_started":1}
I20260927 05:10:52.610970 18379 tablet_service.cc:1467] Processing CreateTablet for tablet d6e62a59145943f0a2ea7091f25323f9 (DEFAULT_TABLE table=test-table [id=e6d0a4a0324a4cf68ec6e2beb06624a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:52.612713 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e62a59145943f0a2ea7091f25323f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:52.612334 18509 tablet_service.cc:1467] Processing CreateTablet for tablet d6e62a59145943f0a2ea7091f25323f9 (DEFAULT_TABLE table=test-table [id=e6d0a4a0324a4cf68ec6e2beb06624a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:52.613899 18509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e62a59145943f0a2ea7091f25323f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:52.618870 18313 heartbeater.cc:503] Master 127.0.97.62:46753 was elected leader, sending a full tablet report...
I20260927 05:10:52.625312 18592 tablet_bootstrap.cc:492] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d: Bootstrap starting.
I20260927 05:10:52.628918 18593 tablet_bootstrap.cc:492] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: Bootstrap starting.
I20260927 05:10:52.630323 18592 tablet_bootstrap.cc:654] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:52.631495 18592 log.cc:824] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:52.632890 18594 tablet_bootstrap.cc:492] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: Bootstrap starting.
I20260927 05:10:52.633579 18593 tablet_bootstrap.cc:654] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:52.634757 18593 log.cc:824] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:52.636363 18592 tablet_bootstrap.cc:492] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d: No bootstrap required, opened a new log
I20260927 05:10:52.636739 18592 ts_tablet_manager.cc:1397] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20260927 05:10:52.636983 18594 tablet_bootstrap.cc:654] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:52.638159 18593 tablet_bootstrap.cc:492] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: No bootstrap required, opened a new log
I20260927 05:10:52.638289 18594 log.cc:824] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:52.638476 18593 ts_tablet_manager.cc:1397] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260927 05:10:52.641886 18594 tablet_bootstrap.cc:492] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: No bootstrap required, opened a new log
I20260927 05:10:52.642223 18594 ts_tablet_manager.cc:1397] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 05:10:52.647212 18593 raft_consensus.cc:348] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 } } peers { permanent_uuid: "f55b48c02ea44f68b6834d44c47d8118" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38335 } } peers { permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 } }
I20260927 05:10:52.647697 18593 raft_consensus.cc:374] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:52.647846 18593 raft_consensus.cc:729] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd040dd0476046b3a4a8ebf3685c7281, State: Initialized, Role: FOLLOWER
I20260927 05:10:52.648666 18593 consensus_queue.cc:260] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 } } peers { permanent_uuid: "f55b48c02ea44f68b6834d44c47d8118" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38335 } } peers { permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 } }
I20260927 05:10:52.648643 18592 raft_consensus.cc:348] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 } } peers { permanent_uuid: "f55b48c02ea44f68b6834d44c47d8118" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38335 } } peers { permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 } }
I20260927 05:10:52.649201 18592 raft_consensus.cc:374] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:52.649394 18592 raft_consensus.cc:729] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704ac6b0db264c1fabbbdb04b15b8e7d, State: Initialized, Role: FOLLOWER
I20260927 05:10:52.650862 18593 ts_tablet_manager.cc:1428] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260927 05:10:52.650959 18574 heartbeater.cc:503] Master 127.0.97.62:46753 was elected leader, sending a full tablet report...
I20260927 05:10:52.650864 18592 consensus_queue.cc:260] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 } } peers { permanent_uuid: "f55b48c02ea44f68b6834d44c47d8118" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38335 } } peers { permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 } }
I20260927 05:10:52.654474 18592 ts_tablet_manager.cc:1428] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260927 05:10:52.655153 18594 raft_consensus.cc:348] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 } } peers { permanent_uuid: "f55b48c02ea44f68b6834d44c47d8118" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38335 } } peers { permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 } }
I20260927 05:10:52.655678 18594 raft_consensus.cc:374] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:52.655814 18594 raft_consensus.cc:729] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55b48c02ea44f68b6834d44c47d8118, State: Initialized, Role: FOLLOWER
I20260927 05:10:52.656622 18594 consensus_queue.cc:260] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 } } peers { permanent_uuid: "f55b48c02ea44f68b6834d44c47d8118" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38335 } } peers { permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 } }
I20260927 05:10:52.657851 18444 heartbeater.cc:503] Master 127.0.97.62:46753 was elected leader, sending a full tablet report...
I20260927 05:10:52.658779 18594 ts_tablet_manager.cc:1428] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260927 05:10:52.759759 18575 tablet.cc:2369] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:10:52.773250 18600 raft_consensus.cc:482] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:10:52.773677 18600 raft_consensus.cc:504] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 } } peers { permanent_uuid: "f55b48c02ea44f68b6834d44c47d8118" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38335 } } peers { permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 } }
I20260927 05:10:52.775516 18600 leader_election.cc:288] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 704ac6b0db264c1fabbbdb04b15b8e7d (127.0.97.1:35461), cd040dd0476046b3a4a8ebf3685c7281 (127.0.97.3:34291)
I20260927 05:10:52.784102 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.776441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56719 (local address 127.0.97.1:35461)
0927 05:10:52.776690 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:52.776697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:52.776883 (+ 186us) server_negotiation.cc:407] Connection header received
0927 05:10:52.777169 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:52.777178 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:52.777279 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:52.777375 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:52.778347 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.779437 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.781679 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.782760 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.782919 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.783037 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:52.783585 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 05:10:52.783741 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 05:10:52.784118 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.776573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36133 (local address 127.0.97.3:34291)
0927 05:10:52.776801 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:52.776809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:52.777239 (+ 430us) server_negotiation.cc:407] Connection header received
0927 05:10:52.777471 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:52.777477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:52.777564 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:52.777675 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:52.778627 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.779670 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.781804 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.782739 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.782921 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.783045 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:52.783641 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 05:10:52.783795 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:10:52.784361 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.775897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35461 (local address 127.0.97.2:56719)
0927 05:10:52.776752 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:52.776777 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:52.776978 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:52.777455 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:52.777461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:52.777492 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:52.778116 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.778147 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.779625 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.779639 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.781425 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.781441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.782988 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.782995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.783274 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.783307 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:10:52.783546 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 05:10:52.783731 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":31,"thread_start_us":414,"threads_started":1}
I20260927 05:10:52.784468 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.775964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34291 (local address 127.0.97.2:36133)
0927 05:10:52.777094 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:52.777112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:52.777266 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:52.777770 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:52.777775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:52.777810 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:52.778436 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.778451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.779839 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.779845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.781552 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:52.781566 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.783038 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.783046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:52.783292 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.783325 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 05:10:52.783613 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 05:10:52.783782 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":30,"thread_start_us":811,"threads_started":1}
I20260927 05:10:52.785713 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e62a59145943f0a2ea7091f25323f9" candidate_uuid: "f55b48c02ea44f68b6834d44c47d8118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" is_pre_election: true
I20260927 05:10:52.786402 18268 raft_consensus.cc:2454] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f55b48c02ea44f68b6834d44c47d8118 in term 0.
I20260927 05:10:52.787286 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e62a59145943f0a2ea7091f25323f9" candidate_uuid: "f55b48c02ea44f68b6834d44c47d8118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd040dd0476046b3a4a8ebf3685c7281" is_pre_election: true
I20260927 05:10:52.787478 18333 leader_election.cc:302] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 704ac6b0db264c1fabbbdb04b15b8e7d, f55b48c02ea44f68b6834d44c47d8118; no voters:
I20260927 05:10:52.788116 18600 raft_consensus.cc:2781] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:10:52.788344 18600 raft_consensus.cc:482] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:10:52.788256 18529 raft_consensus.cc:2454] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f55b48c02ea44f68b6834d44c47d8118 in term 0.
I20260927 05:10:52.788475 18600 raft_consensus.cc:3037] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:52.791241 18600 raft_consensus.cc:504] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 } } peers { permanent_uuid: "f55b48c02ea44f68b6834d44c47d8118" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38335 } } peers { permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 } }
I20260927 05:10:52.791903 18600 leader_election.cc:288] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [CANDIDATE]: Term 1 election: Requested vote from peers 704ac6b0db264c1fabbbdb04b15b8e7d (127.0.97.1:35461), cd040dd0476046b3a4a8ebf3685c7281 (127.0.97.3:34291)
I20260927 05:10:52.792580 18268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e62a59145943f0a2ea7091f25323f9" candidate_uuid: "f55b48c02ea44f68b6834d44c47d8118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d"
I20260927 05:10:52.792793 18529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6e62a59145943f0a2ea7091f25323f9" candidate_uuid: "f55b48c02ea44f68b6834d44c47d8118" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd040dd0476046b3a4a8ebf3685c7281"
I20260927 05:10:52.792877 18268 raft_consensus.cc:3037] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:52.793063 18529 raft_consensus.cc:3037] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:52.796563 18529 raft_consensus.cc:2454] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f55b48c02ea44f68b6834d44c47d8118 in term 1.
I20260927 05:10:52.796563 18268 raft_consensus.cc:2454] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f55b48c02ea44f68b6834d44c47d8118 in term 1.
I20260927 05:10:52.797299 18333 leader_election.cc:302] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 704ac6b0db264c1fabbbdb04b15b8e7d, f55b48c02ea44f68b6834d44c47d8118; no voters:
I20260927 05:10:52.797860 18600 raft_consensus.cc:2781] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:10:52.798486 18600 raft_consensus.cc:686] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 1 LEADER]: Becoming Leader. State: Replica: f55b48c02ea44f68b6834d44c47d8118, State: Running, Role: LEADER
W20260927 05:10:52.799177 18445 tablet.cc:2369] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:10:52.799197 18600 consensus_queue.cc:237] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 } } peers { permanent_uuid: "f55b48c02ea44f68b6834d44c47d8118" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38335 } } peers { permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 } }
I20260927 05:10:52.804849 18127 catalog_manager.cc:5495] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 reported cstate change: term changed from 0 to 1, leader changed from <none> to f55b48c02ea44f68b6834d44c47d8118 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f55b48c02ea44f68b6834d44c47d8118" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f55b48c02ea44f68b6834d44c47d8118" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 } health_report { overall_health: UNKNOWN } } }
I20260927 05:10:52.839687 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.833918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.0.97.2:38335)
0927 05:10:52.834154 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:52.834159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:52.834178 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:10:52.834381 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:52.834385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:52.834483 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:52.834642 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:10:52.834974 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.835742 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.837388 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.837819 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.837979 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.838453 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:10:52.838985 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:10:52.839119 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:52.839272 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 05:10:52.839336 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260927 05:10:52.847321 18314 tablet.cc:2369] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:10:52.848651 18529 raft_consensus.cc:1261] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281 [term 1 FOLLOWER]: Refusing update from remote peer f55b48c02ea44f68b6834d44c47d8118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:10:52.848685 18268 raft_consensus.cc:1261] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d [term 1 FOLLOWER]: Refusing update from remote peer f55b48c02ea44f68b6834d44c47d8118: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:10:52.849699 18603 consensus_queue.cc:1035] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704ac6b0db264c1fabbbdb04b15b8e7d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:10:52.850045 18600 consensus_queue.cc:1035] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd040dd0476046b3a4a8ebf3685c7281" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:10:52.859578 18606 mvcc.cc:205] Tried to move back new op lower bound from 7333830053256536064 to 7333830053070868480. Current Snapshot: MvccSnapshot[applied={T|T < 7333830053256536064}]
I20260927 05:10:52.877478 18127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:10:52.877787 18127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:10:52.882244 18127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb31ae999fd841e5b37509e466e7274b: Sending DeleteTablet for 3 replicas of tablet d6e62a59145943f0a2ea7091f25323f9
I20260927 05:10:52.883797 18248 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e62a59145943f0a2ea7091f25323f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:52 UTC) from {username='slave'} at 127.0.0.1:57546
I20260927 05:10:52.883770 18509 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e62a59145943f0a2ea7091f25323f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:52 UTC) from {username='slave'} at 127.0.0.1:43330
I20260927 05:10:52.884716 18379 tablet_service.cc:1514] Processing DeleteTablet for tablet d6e62a59145943f0a2ea7091f25323f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:52 UTC) from {username='slave'} at 127.0.0.1:51802
I20260927 05:10:52.884831 18615 tablet_replica.cc:330] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: stopping tablet replica
I20260927 05:10:52.885315 18615 raft_consensus.cc:2229] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:10:52.885670 18617 tablet_replica.cc:330] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: stopping tablet replica
I20260927 05:10:52.885995 18615 raft_consensus.cc:2258] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:52.886157 18617 raft_consensus.cc:2229] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:10:52.886595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 18186
I20260927 05:10:52.886878 18617 raft_consensus.cc:2258] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:52.889513 18617 ts_tablet_manager.cc:1905] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:52.897748 18617 ts_tablet_manager.cc:1918] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:52.898092 18617 log.cc:1196] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/d6e62a59145943f0a2ea7091f25323f9
I20260927 05:10:52.898466 18615 ts_tablet_manager.cc:1905] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:52.898841 18617 ts_tablet_manager.cc:1939] T d6e62a59145943f0a2ea7091f25323f9 P f55b48c02ea44f68b6834d44c47d8118: Deleting consensus metadata
I20260927 05:10:52.901196 18113 catalog_manager.cc:4841] TS f55b48c02ea44f68b6834d44c47d8118 (127.0.97.2:38335): tablet d6e62a59145943f0a2ea7091f25323f9 (table test-table [id=e6d0a4a0324a4cf68ec6e2beb06624a1]) successfully deleted
I20260927 05:10:52.905310 18616 tablet_replica.cc:330] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d: stopping tablet replica
I20260927 05:10:52.906127 18616 raft_consensus.cc:2229] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:10:52.906977 18616 raft_consensus.cc:2258] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:52.909538 18616 ts_tablet_manager.cc:1905] T d6e62a59145943f0a2ea7091f25323f9 P 704ac6b0db264c1fabbbdb04b15b8e7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:52.910933 18615 ts_tablet_manager.cc:1918] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:52.911173 18615 log.cc:1196] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/d6e62a59145943f0a2ea7091f25323f9
I20260927 05:10:52.911800 18615 ts_tablet_manager.cc:1939] T d6e62a59145943f0a2ea7091f25323f9 P cd040dd0476046b3a4a8ebf3685c7281: Deleting consensus metadata
I20260927 05:10:52.914016 18588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:52.887445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.0.97.1:35461)
0927 05:10:52.887652 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:52.887660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:52.904719 (+ 17059us) server_negotiation.cc:407] Connection header received
0927 05:10:52.904898 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:52.904908 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:52.905066 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:52.905213 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:52.906004 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.907392 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.909596 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:52.910243 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:52.910362 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:52.912070 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:52.912084 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:52.912102 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:52.913032 (+ 930us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:52.913104 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:52.913131 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:52.913234 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:52.913491 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 05:10:52.913616 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:10:52.913991 18113 catalog_manager.cc:4841] TS cd040dd0476046b3a4a8ebf3685c7281 (127.0.97.3:34291): tablet d6e62a59145943f0a2ea7091f25323f9 (table test-table [id=e6d0a4a0324a4cf68ec6e2beb06624a1]) successfully deleted
I20260927 05:10:52.915509 18308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:53.015015 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 18186
W20260927 05:10:53.034348 18112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35461 (error 108)
I20260927 05:10:53.034529 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 18317
I20260927 05:10:53.037164 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:53.036669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35461 (local address 127.0.0.1:57552)
0927 05:10:53.036831 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:53.036985 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 05:10:53.037629 18112 catalog_manager.cc:4584] TS 704ac6b0db264c1fabbbdb04b15b8e7d (127.0.97.1:35461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6e62a59145943f0a2ea7091f25323f9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35461: connect: Connection refused (error 111)
I20260927 05:10:53.044404 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:53.035756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.0.97.2:38335)
0927 05:10:53.036284 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:53.036293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:53.036971 (+ 678us) server_negotiation.cc:407] Connection header received
0927 05:10:53.037276 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:53.037282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:53.037411 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:53.037495 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:53.038162 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:53.038974 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:53.040490 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:53.040958 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:53.041108 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:53.042326 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:53.042336 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:53.042352 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:53.043270 (+ 918us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:53.043337 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:53.043363 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:53.043530 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:53.043793 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 05:10:53.043946 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362}
I20260927 05:10:53.046094 18439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:53.091665 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:53.091087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35461 (local address 127.0.0.1:57556)
0927 05:10:53.091353 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:53.091453 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 05:10:53.093083 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:53.092697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35461 (local address 127.0.0.1:57566)
0927 05:10:53.092868 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:53.092960 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 05:10:53.153990 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:53.153451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35461 (local address 127.0.0.1:57576)
0927 05:10:53.153697 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:53.153814 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 05:10:53.155221 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:53.154767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35461 (local address 127.0.0.1:57582)
0927 05:10:53.154957 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:53.155060 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 05:10:53.176606 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 18317
I20260927 05:10:53.197065 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 18447
I20260927 05:10:53.204841 18590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:53.198060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.0.97.3:34291)
0927 05:10:53.198308 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:53.198313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:53.198336 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:10:53.198483 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:53.198488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:53.198586 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:53.198702 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:53.199172 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:53.200042 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:53.201483 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:53.201942 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:53.202059 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:53.203240 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:53.203249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:53.203265 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:53.203889 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:53.204031 (+ 142us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:53.204056 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:53.204179 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:53.204319 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 05:10:53.204419 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260927 05:10:53.205674 18569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:53.236119 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:53.235535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35461 (local address 127.0.0.1:57586)
0927 05:10:53.235789 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:53.235878 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 05:10:53.237201 18591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:53.236788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35461 (local address 127.0.0.1:57600)
0927 05:10:53.236992 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:53.237092 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 05:10:53.301059 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 18447
I20260927 05:10:53.318238 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 18096
I20260927 05:10:53.319428 18157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:53.443549 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 18096
2026-09-27T05:10:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2949 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:10:53 dist-test-slave-hgs0 krb5kdc[18626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:10:53 dist-test-slave-hgs0 krb5kdc[18626](info): set up 2 sockets
Sep 27 05:10:53 dist-test-slave-hgs0 krb5kdc[18626](info): commencing operation
krb5kdc: starting...
W20260927 05:10:54.624512 388 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:10:54 dist-test-slave-hgs0 krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:10:54.903694 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37503
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:55.273563 18639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:55.273941 18639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:55.274155 18639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:55.281766 18639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:10:55.281936 18639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:55.282061 18639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:55.282160 18639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:10:55.282246 18639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:10:55.292990 18639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:55.294687 18639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:55.296096 18639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:55.309280 18647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:55.309347 18645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:55.309234 18644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:55.315258 18639 server_base.cc:1034] running on GCE node
I20260927 05:10:55.316431 18639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:55.318192 18639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:55.319474 18639 hybrid_clock.cc:648] HybridClock initialized: now 1790485855319424 us; error 39 us; skew 500 ppm
I20260927 05:10:55.319870 18639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:55 dist-test-slave-hgs0 krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:55.323310 18639 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:10:55.325153 18639 webserver.cc:460] Webserver started at http://127.0.97.62:34463/ using document root <none> and password file <none>
I20260927 05:10:55.325981 18639 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:55.326148 18639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:55.326484 18639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:55.329788 18639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "df31e9e1cf624f67888144477b6b0776"
format_stamp: "Formatted at 2026-09-27 05:10:55 on dist-test-slave-hgs0"
server_key: "66442e8ab3e88d27d3590b1afefc9ed2"
server_key_iv: "5a030f954c42c3a5a7ee51bdbf31690d"
server_key_version: "encryptionkey@0"
I20260927 05:10:55.330649 18639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "df31e9e1cf624f67888144477b6b0776"
format_stamp: "Formatted at 2026-09-27 05:10:55 on dist-test-slave-hgs0"
server_key: "66442e8ab3e88d27d3590b1afefc9ed2"
server_key_iv: "5a030f954c42c3a5a7ee51bdbf31690d"
server_key_version: "encryptionkey@0"
I20260927 05:10:55.336360 18639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:10:55.340265 18654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:55.341882 18639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:10:55.342171 18639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "df31e9e1cf624f67888144477b6b0776"
format_stamp: "Formatted at 2026-09-27 05:10:55 on dist-test-slave-hgs0"
server_key: "66442e8ab3e88d27d3590b1afefc9ed2"
server_key_iv: "5a030f954c42c3a5a7ee51bdbf31690d"
server_key_version: "encryptionkey@0"
I20260927 05:10:55.342433 18639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:55.359776 18639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:55.365022 18639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:55.365384 18639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:55.386402 18639 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37503
I20260927 05:10:55.386421 18705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37503 every 8 connection(s)
I20260927 05:10:55.388234 18639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:10:55.392128 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 18639
I20260927 05:10:55.392349 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:10:55.393007 388 external_mini_cluster.cc:1428] Setting key 4c6e04a099c2a70df9732130d4d6b4f8
I20260927 05:10:55.393194 18706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:55.407176 18706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776: Bootstrap starting.
Sep 27 05:10:55 dist-test-slave-hgs0 krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:55.411538 18706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:55.413232 18706 log.cc:824] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:55.417934 18706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776: No bootstrap required, opened a new log
I20260927 05:10:55.421409 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:55.395180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.0.97.62:37503)
0927 05:10:55.396092 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:55.396116 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:55.396158 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:10:55.398369 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:55.398392 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:55.398791 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:55.399193 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:55.400919 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:55.402248 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:55.403691 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:55.404588 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:55.404831 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:55.409762 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:55.409775 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:55.409798 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:55.409858 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:55.414445 (+ 4587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:55.415636 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:55.415646 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:55.415656 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:55.415914 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:55.416529 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:55.416535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:55.416539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:55.417473 (+ 934us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:55.417839 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:55.419460 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0927 05:10:55.419915 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":418,"threads_started":1}
I20260927 05:10:55.428138 18706 raft_consensus.cc:348] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df31e9e1cf624f67888144477b6b0776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37503 } }
I20260927 05:10:55.428622 18706 raft_consensus.cc:374] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:55.428735 18706 raft_consensus.cc:729] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df31e9e1cf624f67888144477b6b0776, State: Initialized, Role: FOLLOWER
I20260927 05:10:55.429420 18706 consensus_queue.cc:260] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df31e9e1cf624f67888144477b6b0776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37503 } }
I20260927 05:10:55.429697 18706 raft_consensus.cc:388] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:10:55.429853 18706 raft_consensus.cc:482] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:10:55.430045 18706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:55.432811 18706 raft_consensus.cc:504] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df31e9e1cf624f67888144477b6b0776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37503 } }
I20260927 05:10:55.433398 18706 leader_election.cc:302] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df31e9e1cf624f67888144477b6b0776; no voters:
I20260927 05:10:55.434090 18706 leader_election.cc:288] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:10:55.434283 18711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:10:55.435005 18711 raft_consensus.cc:686] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [term 1 LEADER]: Becoming Leader. State: Replica: df31e9e1cf624f67888144477b6b0776, State: Running, Role: LEADER
I20260927 05:10:55.435614 18711 consensus_queue.cc:237] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df31e9e1cf624f67888144477b6b0776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37503 } }
I20260927 05:10:55.436481 18706 sys_catalog.cc:564] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:10:55.438735 18713 sys_catalog.cc:455] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df31e9e1cf624f67888144477b6b0776. Latest consensus state: current_term: 1 leader_uuid: "df31e9e1cf624f67888144477b6b0776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df31e9e1cf624f67888144477b6b0776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37503 } } }
I20260927 05:10:55.439215 18713 sys_catalog.cc:458] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:55.440313 18712 sys_catalog.cc:455] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df31e9e1cf624f67888144477b6b0776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df31e9e1cf624f67888144477b6b0776" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37503 } } }
I20260927 05:10:55.440682 18712 sys_catalog.cc:458] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776 [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:55.441911 18719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:10:55.448159 18719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:10:55.458849 18719 catalog_manager.cc:1347] Generated new cluster ID: be83f73ba08b46e5900fcbe2880611f6
I20260927 05:10:55.458999 18719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:10:55.484323 18719 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:10:55.485819 18719 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:10:55.492390 18719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776: Generated new TSK 0
I20260927 05:10:55.493374 18719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:10:55.567773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37503
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:55.946712 18732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:55.947095 18732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:55.947244 18732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:55.954875 18732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:55.955008 18732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:55.955201 18732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:10:55.965397 18732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:55.966857 18732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:55.968310 18732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:55.983366 18740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:55.983448 18738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:55.983412 18737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:55.984761 18732 server_base.cc:1034] running on GCE node
I20260927 05:10:55.985589 18732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:55.986814 18732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:55.988117 18732 hybrid_clock.cc:648] HybridClock initialized: now 1790485855988037 us; error 71 us; skew 500 ppm
I20260927 05:10:55.988528 18732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:55 dist-test-slave-hgs0 krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:56.000952 18732 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:10:56.002770 18732 webserver.cc:460] Webserver started at http://127.0.97.1:45765/ using document root <none> and password file <none>
I20260927 05:10:56.003573 18732 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:56.003746 18732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:56.004169 18732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:56.007458 18732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "be7528d7d9714a0eab95b0b6e2c6dc75"
format_stamp: "Formatted at 2026-09-27 05:10:56 on dist-test-slave-hgs0"
server_key: "054a0319cb1edddfbdd15c20dccea347"
server_key_iv: "ed63105fcc40ec350d564b3ecae5e139"
server_key_version: "encryptionkey@0"
I20260927 05:10:56.008407 18732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "be7528d7d9714a0eab95b0b6e2c6dc75"
format_stamp: "Formatted at 2026-09-27 05:10:56 on dist-test-slave-hgs0"
server_key: "054a0319cb1edddfbdd15c20dccea347"
server_key_iv: "ed63105fcc40ec350d564b3ecae5e139"
server_key_version: "encryptionkey@0"
I20260927 05:10:56.014394 18732 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:10:56.018743 18747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:56.020434 18732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:10:56.020632 18732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "be7528d7d9714a0eab95b0b6e2c6dc75"
format_stamp: "Formatted at 2026-09-27 05:10:56 on dist-test-slave-hgs0"
server_key: "054a0319cb1edddfbdd15c20dccea347"
server_key_iv: "ed63105fcc40ec350d564b3ecae5e139"
server_key_version: "encryptionkey@0"
I20260927 05:10:56.020874 18732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:56.041186 18732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:56.046084 18732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:56.046468 18732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:56.047662 18732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:56.049360 18732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:56.049474 18732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:56.049670 18732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:56.049746 18732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:56.086833 18732 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33527
I20260927 05:10:56.086856 18859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33527 every 8 connection(s)
I20260927 05:10:56.088274 18732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:10:56.090159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 18732
I20260927 05:10:56.090339 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:10:56.090884 388 external_mini_cluster.cc:1428] Setting key 2f602933e134f7f597fb760af6e4896d
Sep 27 05:10:56 dist-test-slave-hgs0 krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:56.115365 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:56.091476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48437 (local address 127.0.97.62:37503)
0927 05:10:56.092085 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:56.092091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:56.093547 (+ 1456us) server_negotiation.cc:407] Connection header received
0927 05:10:56.095780 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:56.095789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:56.095899 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:56.096062 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:56.098181 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:56.099489 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:56.101620 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:56.102104 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:56.102307 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:56.106993 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:56.107004 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:56.107012 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:56.107077 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:56.111163 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:56.112318 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:56.112328 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:56.112335 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:56.112521 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:56.113113 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:56.113121 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:56.113127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:56.113581 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:56.113757 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:56.114597 (+ 840us) server_negotiation.cc:299] Negotiation successful
0927 05:10:56.114909 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":306,"threads_started":1}
I20260927 05:10:56.116529 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:56.092284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37503 (local address 127.0.97.1:48437)
0927 05:10:56.093321 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:56.093375 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:56.095429 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:56.096375 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:56.096386 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:56.097037 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:56.097756 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:56.097940 (+ 184us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:56.099747 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:56.099751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:56.101412 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:56.101429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:56.102423 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:56.102432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:56.102760 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:56.103583 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:56.103610 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:56.106678 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:56.111408 (+ 4730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:56.111436 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:56.111453 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:56.112139 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:56.112693 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:56.112697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:56.112700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:56.112951 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:56.113739 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:56.113747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:56.114145 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 05:10:56.114407 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 05:10:56.114976 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":455,"threads_started":1}
I20260927 05:10:56.118597 18860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37503
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260927 05:10:56.118995 18860 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:56.120038 18860 heartbeater.cc:511] Master 127.0.97.62:37503 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260927 05:10:56.123013 18671 ts_manager.cc:194] Registered new tserver with Master: be7528d7d9714a0eab95b0b6e2c6dc75 (127.0.97.1:33527)
I20260927 05:10:56.124907 18671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:10:56.156694 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37503
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:56.516847 18866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:56.517177 18866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:56.517282 18866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:56.525089 18866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:56.525213 18866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:56.525329 18866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:10:56.535701 18866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:56.537412 18866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:56.538928 18866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:56.554529 18871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:56.554620 18872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:56.554668 18874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:56.556051 18866 server_base.cc:1034] running on GCE node
I20260927 05:10:56.556811 18866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:56.557976 18866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:56.559268 18866 hybrid_clock.cc:648] HybridClock initialized: now 1790485856559212 us; error 49 us; skew 500 ppm
I20260927 05:10:56.559695 18866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:56 dist-test-slave-hgs0 krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:56.563457 18866 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:10:56.565241 18866 webserver.cc:460] Webserver started at http://127.0.97.2:34339/ using document root <none> and password file <none>
I20260927 05:10:56.566078 18866 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:56.566242 18866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:56.566604 18866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:56.570013 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "cd49cbca186a4af49f2ac83f37616020"
format_stamp: "Formatted at 2026-09-27 05:10:56 on dist-test-slave-hgs0"
server_key: "f8c5b1e8095aae03afc8980abe7d8502"
server_key_iv: "ad44d01a4ad0584dde37659cf16bef7c"
server_key_version: "encryptionkey@0"
I20260927 05:10:56.570946 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cd49cbca186a4af49f2ac83f37616020"
format_stamp: "Formatted at 2026-09-27 05:10:56 on dist-test-slave-hgs0"
server_key: "f8c5b1e8095aae03afc8980abe7d8502"
server_key_iv: "ad44d01a4ad0584dde37659cf16bef7c"
server_key_version: "encryptionkey@0"
I20260927 05:10:56.577790 18866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260927 05:10:56.582350 18881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:56.584271 18866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:10:56.584504 18866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "cd49cbca186a4af49f2ac83f37616020"
format_stamp: "Formatted at 2026-09-27 05:10:56 on dist-test-slave-hgs0"
server_key: "f8c5b1e8095aae03afc8980abe7d8502"
server_key_iv: "ad44d01a4ad0584dde37659cf16bef7c"
server_key_version: "encryptionkey@0"
I20260927 05:10:56.584757 18866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:56.603387 18866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:56.608405 18866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:56.608729 18866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:56.609784 18866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:56.611230 18866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:56.611306 18866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:56.611512 18866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:56.611563 18866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:56.647763 18866 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43473
I20260927 05:10:56.647770 18993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43473 every 8 connection(s)
I20260927 05:10:56.649187 18866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:10:56.654536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 18866
I20260927 05:10:56.654748 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:10:56.655198 388 external_mini_cluster.cc:1428] Setting key d2ef9bc22370842985e2b2209457af28
Sep 27 05:10:56 dist-test-slave-hgs0 krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:56.672374 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:56.651982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44921 (local address 127.0.97.62:37503)
0927 05:10:56.652444 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:56.652449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:56.653644 (+ 1195us) server_negotiation.cc:407] Connection header received
0927 05:10:56.655825 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:56.655831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:56.655982 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:56.656117 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:56.657762 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:56.658531 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:56.660245 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:56.660727 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:56.660866 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:56.665123 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:56.665129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:56.665133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:56.665176 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:56.668372 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:56.669397 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:56.669402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:56.669405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:56.669571 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:56.670166 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:56.670171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:56.670173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:56.670504 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:56.670702 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:56.671603 (+ 901us) server_negotiation.cc:299] Negotiation successful
0927 05:10:56.671886 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":232,"threads_started":1}
I20260927 05:10:56.673592 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:56.652621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37503 (local address 127.0.97.2:44921)
0927 05:10:56.653426 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:56.653475 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:56.655500 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:56.656395 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:56.656404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:56.656957 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:56.657580 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:56.657599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:56.658701 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:56.658711 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:56.660088 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:56.660101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:56.660968 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:56.660975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:56.661236 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:56.662003 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:56.662032 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:56.664775 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:56.668594 (+ 3819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:56.668607 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:56.668640 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:56.669203 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:56.669720 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:56.669726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:56.669730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:56.669987 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:56.670671 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:56.670681 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:56.671149 (+ 468us) client_negotiation.cc:769] Sending connection context
0927 05:10:56.671470 (+ 321us) client_negotiation.cc:240] Negotiation successful
0927 05:10:56.672183 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":323,"threads_started":1}
I20260927 05:10:56.675560 18994 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37503
I20260927 05:10:56.675997 18994 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:56.676940 18994 heartbeater.cc:511] Master 127.0.97.62:37503 requested a full tablet report, sending...
I20260927 05:10:56.678954 18671 ts_manager.cc:194] Registered new tserver with Master: cd49cbca186a4af49f2ac83f37616020 (127.0.97.2:43473)
I20260927 05:10:56.680347 18671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44921
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:10:56.720469 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37503
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:10:57.087867 19000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:57.088271 19000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:57.088466 19000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:57.097335 19000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:57.097472 19000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:57.097666 19000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:10:57.108637 19000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:57.110097 19000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:57.111577 19000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:57.127450 19005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:57.127542 19008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:57.127512 19006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:57.128511 19000 server_base.cc:1034] running on GCE node
I20260927 05:10:57.128832 18860 heartbeater.cc:503] Master 127.0.97.62:37503 was elected leader, sending a full tablet report...
I20260927 05:10:57.129297 19000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:57.130508 19000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:57.131764 19000 hybrid_clock.cc:648] HybridClock initialized: now 1790485857131686 us; error 75 us; skew 500 ppm
I20260927 05:10:57.132189 19000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:10:57 dist-test-slave-hgs0 krb5kdc[18626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:10:57.135660 19000 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:10:57.138348 19000 webserver.cc:460] Webserver started at http://127.0.97.3:41741/ using document root <none> and password file <none>
I20260927 05:10:57.139489 19000 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:57.139700 19000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:57.140126 19000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:57.143738 19000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62"
format_stamp: "Formatted at 2026-09-27 05:10:57 on dist-test-slave-hgs0"
server_key: "84f1a3befb6b2258fde11ccd29a9a5db"
server_key_iv: "92d5b1cf5239e9b6582caa2cfea29b3e"
server_key_version: "encryptionkey@0"
I20260927 05:10:57.144817 19000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62"
format_stamp: "Formatted at 2026-09-27 05:10:57 on dist-test-slave-hgs0"
server_key: "84f1a3befb6b2258fde11ccd29a9a5db"
server_key_iv: "92d5b1cf5239e9b6582caa2cfea29b3e"
server_key_version: "encryptionkey@0"
I20260927 05:10:57.152036 19000 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 05:10:57.156594 19015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:57.158465 19000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260927 05:10:57.158707 19000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62"
format_stamp: "Formatted at 2026-09-27 05:10:57 on dist-test-slave-hgs0"
server_key: "84f1a3befb6b2258fde11ccd29a9a5db"
server_key_iv: "92d5b1cf5239e9b6582caa2cfea29b3e"
server_key_version: "encryptionkey@0"
I20260927 05:10:57.158950 19000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:57.179234 19000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:57.184756 19000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:57.185112 19000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:57.186335 19000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:57.188030 19000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:57.188143 19000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 05:10:57.188359 19000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:57.188417 19000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:57.227447 19000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39859
I20260927 05:10:57.227504 19127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39859 every 8 connection(s)
I20260927 05:10:57.229027 19000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:10:57.239246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 19000
I20260927 05:10:57.239481 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:10:57.240041 388 external_mini_cluster.cc:1428] Setting key aedb8994d1410872d7cb36e703838ff1
Sep 27 05:10:57 dist-test-slave-hgs0 krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:10:57.254180 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.232369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42577 (local address 127.0.97.62:37503)
0927 05:10:57.232970 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.232976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.234048 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 05:10:57.236387 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.236393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.236525 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.236674 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:57.238606 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.239481 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.241513 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.242185 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.242346 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.246809 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:57.246815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:57.246819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:57.246891 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:57.250226 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:57.251240 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:57.251247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:57.251251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:57.251429 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:57.252019 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:57.252024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:57.252026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:57.252350 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:57.252537 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.253437 (+ 900us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.253742 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":322,"threads_started":1}
I20260927 05:10:57.255657 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.232917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37503 (local address 127.0.97.3:42577)
0927 05:10:57.233807 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:57.233864 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:57.236005 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:57.237023 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:57.237033 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:57.237673 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:57.238358 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.238380 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.239660 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.239665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.241326 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.241343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.242493 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.242502 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.242784 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.243606 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:10:57.243640 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:57.246453 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:57.250447 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:57.250459 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:57.250482 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:57.251064 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:57.251571 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:10:57.251577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:10:57.251581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:10:57.251834 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:10:57.252526 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:57.252536 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:57.252978 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 05:10:57.253317 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 05:10:57.254008 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":366,"threads_started":1}
I20260927 05:10:57.257369 19128 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37503
I20260927 05:10:57.257783 19128 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:57.258665 19128 heartbeater.cc:511] Master 127.0.97.62:37503 requested a full tablet report, sending...
I20260927 05:10:57.260502 18671 ts_manager.cc:194] Registered new tserver with Master: 96dfd0eb1b4741d3a7f378fa4f0e8f62 (127.0.97.3:39859)
I20260927 05:10:57.261394 18671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42577
I20260927 05:10:57.269712 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:10:57.290261 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.274344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.0.97.62:37503)
0927 05:10:57.274602 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.274613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.275125 (+ 512us) server_negotiation.cc:407] Connection header received
0927 05:10:57.275270 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.275275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.275354 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.275457 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:57.277047 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.277852 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.279278 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.279700 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.279805 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.282965 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:57.282971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:57.282975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:57.283031 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:57.286236 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:57.287445 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:57.287450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:57.287453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:57.287638 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:57.288261 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:57.288266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:57.288268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:57.288822 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:57.289126 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.289575 (+ 449us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.289831 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:10:57.295182 18671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:10:57.303259 18671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:10:57.334179 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.318706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.0.97.2:43473)
0927 05:10:57.321933 (+ 3227us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.321945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.321978 (+ 33us) server_negotiation.cc:407] Connection header received
0927 05:10:57.322112 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.322120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.322475 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.323783 (+ 1308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:57.324179 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.326474 (+ 2295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.330703 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.332472 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.333237 (+ 765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.333503 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.333588 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.333812 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3050,"thread_start_us":2942,"threads_started":1}
I20260927 05:10:57.334280 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.319069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43473 (local address 127.0.0.1:51324)
0927 05:10:57.321256 (+ 2187us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:57.321277 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:57.321384 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:57.322817 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:57.322829 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:57.322875 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:57.323469 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.323487 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.326742 (+ 3255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.326753 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.330309 (+ 3556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.330341 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.332754 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.332763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.333065 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.333289 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 05:10:57.333461 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 05:10:57.333699 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2050,"mutex_wait_us":83,"thread_start_us":1904,"threads_started":1}
I20260927 05:10:57.339233 18929 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc69cec2d47484b8b3bc1d01c241e79 (DEFAULT_TABLE table=test-table [id=4421fffd71a04ba9bca2bec839680eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:57.341104 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.322146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33527 (local address 127.0.0.1:41302)
0927 05:10:57.331657 (+ 9511us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:57.331681 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:57.331845 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:57.332835 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:57.332839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:57.332874 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:57.333380 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.333394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.335758 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.335763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.338254 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.338271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.340175 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.340184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.340364 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.340401 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:10:57.340477 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 05:10:57.340646 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9352,"mutex_wait_us":80,"thread_start_us":9345,"threads_started":1}
I20260927 05:10:57.341210 18929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc69cec2d47484b8b3bc1d01c241e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:57.341665 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.318108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.0.97.3:39859)
0927 05:10:57.333168 (+ 15060us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.333180 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.333222 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:10:57.333339 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.333347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.333611 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.334879 (+ 1268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:57.335208 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.336704 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.339171 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.340467 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.340625 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.340918 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.341141 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.341306 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14815,"thread_start_us":14653,"threads_started":1}
I20260927 05:10:57.341645 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.317860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39859 (local address 127.0.0.1:59970)
0927 05:10:57.320299 (+ 2439us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:57.320355 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:57.320608 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:57.333864 (+ 13256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:57.333876 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:57.333909 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:57.334324 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.334341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.336899 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.336904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.338979 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.338998 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.340707 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.340715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.340900 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.340940 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 05:10:57.341084 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 05:10:57.341231 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2112,"mutex_wait_us":33,"thread_start_us":1198,"threads_started":1}
I20260927 05:10:57.342437 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.321742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.0.97.1:33527)
0927 05:10:57.331444 (+ 9702us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.331459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.332009 (+ 550us) server_negotiation.cc:407] Connection header received
0927 05:10:57.332148 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.332161 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.332593 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.333615 (+ 1022us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:57.333888 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.335585 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.338438 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.339908 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.341562 (+ 1654us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.341830 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.341906 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.342083 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9416,"thread_start_us":9168,"threads_started":1}
I20260927 05:10:57.345652 19063 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc69cec2d47484b8b3bc1d01c241e79 (DEFAULT_TABLE table=test-table [id=4421fffd71a04ba9bca2bec839680eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:57.346410 18795 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc69cec2d47484b8b3bc1d01c241e79 (DEFAULT_TABLE table=test-table [id=4421fffd71a04ba9bca2bec839680eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:10:57.347040 19063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc69cec2d47484b8b3bc1d01c241e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:57.348004 18795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc69cec2d47484b8b3bc1d01c241e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:57.357407 19147 tablet_bootstrap.cc:492] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: Bootstrap starting.
I20260927 05:10:57.359817 19148 tablet_bootstrap.cc:492] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: Bootstrap starting.
I20260927 05:10:57.361888 19147 tablet_bootstrap.cc:654] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:57.363983 19147 log.cc:824] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:57.364147 19148 tablet_bootstrap.cc:654] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:57.365942 19148 log.cc:824] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:57.367790 19149 tablet_bootstrap.cc:492] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: Bootstrap starting.
I20260927 05:10:57.370577 19148 tablet_bootstrap.cc:492] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: No bootstrap required, opened a new log
I20260927 05:10:57.370949 19148 ts_tablet_manager.cc:1397] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 05:10:57.371793 19147 tablet_bootstrap.cc:492] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: No bootstrap required, opened a new log
I20260927 05:10:57.372143 19147 ts_tablet_manager.cc:1397] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260927 05:10:57.372612 19149 tablet_bootstrap.cc:654] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:57.373984 19149 log.cc:824] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:57.377713 19149 tablet_bootstrap.cc:492] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: No bootstrap required, opened a new log
I20260927 05:10:57.378065 19149 ts_tablet_manager.cc:1397] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 05:10:57.380775 19147 raft_consensus.cc:348] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 } } peers { permanent_uuid: "cd49cbca186a4af49f2ac83f37616020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43473 } } peers { permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 } }
I20260927 05:10:57.381222 19147 raft_consensus.cc:374] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:57.381325 19147 raft_consensus.cc:729] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd49cbca186a4af49f2ac83f37616020, State: Initialized, Role: FOLLOWER
I20260927 05:10:57.382053 19147 consensus_queue.cc:260] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 } } peers { permanent_uuid: "cd49cbca186a4af49f2ac83f37616020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43473 } } peers { permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 } }
I20260927 05:10:57.382010 19148 raft_consensus.cc:348] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 } } peers { permanent_uuid: "cd49cbca186a4af49f2ac83f37616020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43473 } } peers { permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 } }
I20260927 05:10:57.382490 19148 raft_consensus.cc:374] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:57.382608 19148 raft_consensus.cc:729] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96dfd0eb1b4741d3a7f378fa4f0e8f62, State: Initialized, Role: FOLLOWER
I20260927 05:10:57.383445 18994 heartbeater.cc:503] Master 127.0.97.62:37503 was elected leader, sending a full tablet report...
I20260927 05:10:57.383381 19148 consensus_queue.cc:260] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 } } peers { permanent_uuid: "cd49cbca186a4af49f2ac83f37616020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43473 } } peers { permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 } }
I20260927 05:10:57.389485 19147 ts_tablet_manager.cc:1428] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: Time spent starting tablet: real 0.017s user 0.008s sys 0.005s
I20260927 05:10:57.392443 19149 raft_consensus.cc:348] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 } } peers { permanent_uuid: "cd49cbca186a4af49f2ac83f37616020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43473 } } peers { permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 } }
I20260927 05:10:57.393029 19149 raft_consensus.cc:374] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:57.393162 19149 raft_consensus.cc:729] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7528d7d9714a0eab95b0b6e2c6dc75, State: Initialized, Role: FOLLOWER
I20260927 05:10:57.393258 19128 heartbeater.cc:503] Master 127.0.97.62:37503 was elected leader, sending a full tablet report...
I20260927 05:10:57.394199 19148 ts_tablet_manager.cc:1428] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: Time spent starting tablet: real 0.023s user 0.024s sys 0.000s
I20260927 05:10:57.394011 19149 consensus_queue.cc:260] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 } } peers { permanent_uuid: "cd49cbca186a4af49f2ac83f37616020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43473 } } peers { permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 } }
I20260927 05:10:57.396399 19149 ts_tablet_manager.cc:1428] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: Time spent starting tablet: real 0.018s user 0.020s sys 0.000s
W20260927 05:10:57.401315 18995 tablet.cc:2369] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:10:57.413800 19153 raft_consensus.cc:482] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:10:57.414222 19153 raft_consensus.cc:504] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 } } peers { permanent_uuid: "cd49cbca186a4af49f2ac83f37616020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43473 } } peers { permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 } }
I20260927 05:10:57.416373 19153 leader_election.cc:288] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96dfd0eb1b4741d3a7f378fa4f0e8f62 (127.0.97.3:39859), be7528d7d9714a0eab95b0b6e2c6dc75 (127.0.97.1:33527)
I20260927 05:10:57.423995 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.416900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40213 (local address 127.0.97.3:39859)
0927 05:10:57.417348 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.417359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.418131 (+ 772us) server_negotiation.cc:407] Connection header received
0927 05:10:57.418323 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.418350 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.418436 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.418549 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:57.419196 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.420057 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.422030 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.422841 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.422977 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.423109 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.423521 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.423676 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260927 05:10:57.423992 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.417019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44251 (local address 127.0.97.1:33527)
0927 05:10:57.417289 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.417300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.418053 (+ 753us) server_negotiation.cc:407] Connection header received
0927 05:10:57.418359 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.418363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.418472 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.418566 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:10:57.419222 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.420047 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.422028 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.422852 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.423026 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.423151 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.423518 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.423676 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:10:57.424301 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.416907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39859 (local address 127.0.97.2:40213)
0927 05:10:57.418038 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:57.418053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:57.418178 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:57.418580 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:57.418593 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:57.418619 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:57.419018 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.419030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.420193 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.420199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.421857 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.421871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.423105 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.423116 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.423328 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.423371 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 05:10:57.423490 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 05:10:57.423715 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":85,"thread_start_us":624,"threads_started":1}
I20260927 05:10:57.424681 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.416625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33527 (local address 127.0.97.2:44251)
0927 05:10:57.417935 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:57.417952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:57.418132 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:57.418618 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:57.418624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:57.418647 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:10:57.419056 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.419068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.420205 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.420212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.421856 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:57.421871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.423131 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.423149 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:57.423331 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.423373 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 05:10:57.423488 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 05:10:57.423702 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"mutex_wait_us":134,"thread_start_us":1085,"threads_started":1}
I20260927 05:10:57.425536 19083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc69cec2d47484b8b3bc1d01c241e79" candidate_uuid: "cd49cbca186a4af49f2ac83f37616020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" is_pre_election: true
I20260927 05:10:57.425776 18815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc69cec2d47484b8b3bc1d01c241e79" candidate_uuid: "cd49cbca186a4af49f2ac83f37616020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" is_pre_election: true
I20260927 05:10:57.426136 19083 raft_consensus.cc:2454] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd49cbca186a4af49f2ac83f37616020 in term 0.
I20260927 05:10:57.426404 18815 raft_consensus.cc:2454] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd49cbca186a4af49f2ac83f37616020 in term 0.
I20260927 05:10:57.426923 18882 leader_election.cc:302] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96dfd0eb1b4741d3a7f378fa4f0e8f62, cd49cbca186a4af49f2ac83f37616020; no voters:
I20260927 05:10:57.427371 19153 raft_consensus.cc:2781] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:10:57.427543 19153 raft_consensus.cc:482] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:10:57.427654 19153 raft_consensus.cc:3037] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:57.430047 19153 raft_consensus.cc:504] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 } } peers { permanent_uuid: "cd49cbca186a4af49f2ac83f37616020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43473 } } peers { permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 } }
I20260927 05:10:57.430824 19153 leader_election.cc:288] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [CANDIDATE]: Term 1 election: Requested vote from peers 96dfd0eb1b4741d3a7f378fa4f0e8f62 (127.0.97.3:39859), be7528d7d9714a0eab95b0b6e2c6dc75 (127.0.97.1:33527)
I20260927 05:10:57.431334 19083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc69cec2d47484b8b3bc1d01c241e79" candidate_uuid: "cd49cbca186a4af49f2ac83f37616020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62"
I20260927 05:10:57.431459 18815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc69cec2d47484b8b3bc1d01c241e79" candidate_uuid: "cd49cbca186a4af49f2ac83f37616020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75"
I20260927 05:10:57.431566 19083 raft_consensus.cc:3037] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:57.431725 18815 raft_consensus.cc:3037] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:57.434265 18815 raft_consensus.cc:2454] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd49cbca186a4af49f2ac83f37616020 in term 1.
I20260927 05:10:57.434283 19083 raft_consensus.cc:2454] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd49cbca186a4af49f2ac83f37616020 in term 1.
I20260927 05:10:57.435207 18885 leader_election.cc:302] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be7528d7d9714a0eab95b0b6e2c6dc75, cd49cbca186a4af49f2ac83f37616020; no voters:
I20260927 05:10:57.435571 19153 raft_consensus.cc:2781] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:10:57.436136 19153 raft_consensus.cc:686] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 1 LEADER]: Becoming Leader. State: Replica: cd49cbca186a4af49f2ac83f37616020, State: Running, Role: LEADER
I20260927 05:10:57.436682 19153 consensus_queue.cc:237] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 } } peers { permanent_uuid: "cd49cbca186a4af49f2ac83f37616020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43473 } } peers { permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 } }
I20260927 05:10:57.442373 18669 catalog_manager.cc:5495] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd49cbca186a4af49f2ac83f37616020 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "cd49cbca186a4af49f2ac83f37616020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd49cbca186a4af49f2ac83f37616020" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 } health_report { overall_health: UNKNOWN } } }
I20260927 05:10:57.474634 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.468343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.0.97.2:43473)
0927 05:10:57.468624 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.468630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.468655 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:10:57.468810 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.468814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.468901 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.469078 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:10:57.469508 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.470266 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.471919 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.472508 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.472672 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.473327 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:10:57.473890 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:10:57.474107 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.474293 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.474346 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 05:10:57.481024 19129 tablet.cc:2369] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:10:57.484369 19083 raft_consensus.cc:1261] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62 [term 1 FOLLOWER]: Refusing update from remote peer cd49cbca186a4af49f2ac83f37616020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:10:57.484369 18815 raft_consensus.cc:1261] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75 [term 1 FOLLOWER]: Refusing update from remote peer cd49cbca186a4af49f2ac83f37616020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:10:57.485564 19158 consensus_queue.cc:1035] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96dfd0eb1b4741d3a7f378fa4f0e8f62" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:10:57.485945 19153 consensus_queue.cc:1035] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be7528d7d9714a0eab95b0b6e2c6dc75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:10:57.501633 19161 mvcc.cc:205] Tried to move back new op lower bound from 7333830072244129792 to 7333830072065458176. Current Snapshot: MvccSnapshot[applied={T|T < 7333830072244129792}]
I20260927 05:10:57.521297 18669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:10:57.521638 18669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:10:57.526238 18669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df31e9e1cf624f67888144477b6b0776: Sending DeleteTablet for 3 replicas of tablet 7cc69cec2d47484b8b3bc1d01c241e79
I20260927 05:10:57.527848 18795 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cc69cec2d47484b8b3bc1d01c241e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41302
I20260927 05:10:57.527858 19063 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cc69cec2d47484b8b3bc1d01c241e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59970
I20260927 05:10:57.528090 18929 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cc69cec2d47484b8b3bc1d01c241e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:10:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51324
I20260927 05:10:57.534354 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 18732
I20260927 05:10:57.540560 19171 tablet_replica.cc:330] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: stopping tablet replica
I20260927 05:10:57.541206 19171 raft_consensus.cc:2229] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:10:57.541501 19173 tablet_replica.cc:330] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: stopping tablet replica
I20260927 05:10:57.541867 19171 raft_consensus.cc:2258] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:57.541976 19173 raft_consensus.cc:2229] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:10:57.542711 19173 raft_consensus.cc:2258] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:57.542979 19172 tablet_replica.cc:330] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: stopping tablet replica
I20260927 05:10:57.543432 19172 raft_consensus.cc:2229] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:10:57.544049 19172 raft_consensus.cc:2258] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:10:57.545226 19173 ts_tablet_manager.cc:1905] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:57.545831 19171 ts_tablet_manager.cc:1905] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:57.546267 19172 ts_tablet_manager.cc:1905] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:10:57.553212 19173 ts_tablet_manager.cc:1918] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:57.553416 19173 log.cc:1196] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/7cc69cec2d47484b8b3bc1d01c241e79
I20260927 05:10:57.553958 19173 ts_tablet_manager.cc:1939] T 7cc69cec2d47484b8b3bc1d01c241e79 P cd49cbca186a4af49f2ac83f37616020: Deleting consensus metadata
I20260927 05:10:57.554307 19172 ts_tablet_manager.cc:1918] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:57.554494 19172 log.cc:1196] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/7cc69cec2d47484b8b3bc1d01c241e79
I20260927 05:10:57.554659 19171 ts_tablet_manager.cc:1918] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:10:57.554908 19171 log.cc:1196] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/7cc69cec2d47484b8b3bc1d01c241e79
I20260927 05:10:57.555102 19172 ts_tablet_manager.cc:1939] T 7cc69cec2d47484b8b3bc1d01c241e79 P be7528d7d9714a0eab95b0b6e2c6dc75: Deleting consensus metadata
I20260927 05:10:57.555598 19171 ts_tablet_manager.cc:1939] T 7cc69cec2d47484b8b3bc1d01c241e79 P 96dfd0eb1b4741d3a7f378fa4f0e8f62: Deleting consensus metadata
I20260927 05:10:57.556083 18658 catalog_manager.cc:4841] TS cd49cbca186a4af49f2ac83f37616020 (127.0.97.2:43473): tablet 7cc69cec2d47484b8b3bc1d01c241e79 (table test-table [id=4421fffd71a04ba9bca2bec839680eec]) successfully deleted
I20260927 05:10:57.556942 18658 catalog_manager.cc:4841] TS be7528d7d9714a0eab95b0b6e2c6dc75 (127.0.97.1:33527): tablet 7cc69cec2d47484b8b3bc1d01c241e79 (table test-table [id=4421fffd71a04ba9bca2bec839680eec]) successfully deleted
I20260927 05:10:57.557423 18655 catalog_manager.cc:4841] TS 96dfd0eb1b4741d3a7f378fa4f0e8f62 (127.0.97.3:39859): tablet 7cc69cec2d47484b8b3bc1d01c241e79 (table test-table [id=4421fffd71a04ba9bca2bec839680eec]) successfully deleted
Sep 27 05:10:57 dist-test-slave-hgs0 krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:10:57.572153 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.536644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.0.97.1:33527)
0927 05:10:57.536962 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.536989 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.553753 (+ 16764us) server_negotiation.cc:407] Connection header received
0927 05:10:57.554324 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.554334 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.554438 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.554525 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:57.557266 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.558134 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.560102 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.560486 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.560599 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.565905 (+ 5306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:57.565921 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:57.565937 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:57.565983 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:57.569021 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:57.569859 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:57.569867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:57.569870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:57.570011 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:57.570441 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:57.570445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:57.570447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:57.570828 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:57.571158 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.571472 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.571729 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260927 05:10:57.572814 18855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:57.702884 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 18732
I20260927 05:10:57.728693 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 18866
Sep 27 05:10:57 dist-test-slave-hgs0 krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:10:57.747071 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.729698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.0.97.2:43473)
0927 05:10:57.729916 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.729924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.729946 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:10:57.730013 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.730017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.730079 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.730210 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:57.731546 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.732645 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.734021 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.734503 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.734662 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.739991 (+ 5329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:57.740016 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:57.740034 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:57.740077 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:57.743747 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:57.744661 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:57.744679 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:57.744682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:57.744846 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:57.745292 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:57.745298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:57.745306 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:57.745711 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:57.746143 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.746430 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.746697 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:10:57.748176 18989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:57.871006 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 18866
I20260927 05:10:57.890867 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 19000
Sep 27 05:10:57 dist-test-slave-hgs0 krb5kdc[18626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:10:57.908882 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:57.891709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.0.97.3:39859)
0927 05:10:57.891917 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:57.891925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:57.891989 (+ 64us) server_negotiation.cc:407] Connection header received
0927 05:10:57.892314 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:57.892319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:57.892392 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:57.892532 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:57.893986 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.894873 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.896592 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:57.897054 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:57.897211 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:57.902271 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:57.902290 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:57.902307 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:10:57.902361 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:57.905352 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:57.906309 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:57.906318 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:57.906321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:57.906477 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:10:57.907001 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:10:57.907006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:10:57.907009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:10:57.907436 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:57.907811 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:57.908204 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 05:10:57.908460 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:10:57.909690 19123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:58.005245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 19000
I20260927 05:10:58.023348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 18639
I20260927 05:10:58.024544 18701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:10:58.141140 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 18639
2026-09-27T05:10:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4701 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T05:10:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:10:58Z Disabled control of system clock
I20260927 05:10:58.227193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39333
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:10:58.586102 19182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:58.586447 19182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:58.586588 19182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:58.594841 19182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:10:58.594985 19182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:58.595036 19182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:58.595135 19182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:10:58.595206 19182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:10:58.605429 19182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39333
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:58.606869 19182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:58.608844 19182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:58.621872 19187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:58.621966 19190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:58.621912 19188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:58.622958 19182 server_base.cc:1034] running on GCE node
I20260927 05:10:58.623976 19182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:58.625684 19182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:58.626984 19182 hybrid_clock.cc:648] HybridClock initialized: now 1790485858626921 us; error 55 us; skew 500 ppm
I20260927 05:10:58.627555 19182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:10:58.630532 19182 webserver.cc:460] Webserver started at http://127.0.97.62:42213/ using document root <none> and password file <none>
I20260927 05:10:58.631376 19182 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:58.631536 19182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:58.631907 19182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:58.635303 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "354dab88871a4f22a75b0785d9f9299e"
format_stamp: "Formatted at 2026-09-27 05:10:58 on dist-test-slave-hgs0"
server_key: "30c501155c6123c4a4a07949ceefb563"
server_key_iv: "c6365692b50103d3be34ae4530cb9d5f"
server_key_version: "encryptionkey@0"
I20260927 05:10:58.636309 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "354dab88871a4f22a75b0785d9f9299e"
format_stamp: "Formatted at 2026-09-27 05:10:58 on dist-test-slave-hgs0"
server_key: "30c501155c6123c4a4a07949ceefb563"
server_key_iv: "c6365692b50103d3be34ae4530cb9d5f"
server_key_version: "encryptionkey@0"
I20260927 05:10:58.642572 19182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:10:58.646874 19196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:58.648571 19182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 05:10:58.648797 19182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "354dab88871a4f22a75b0785d9f9299e"
format_stamp: "Formatted at 2026-09-27 05:10:58 on dist-test-slave-hgs0"
server_key: "30c501155c6123c4a4a07949ceefb563"
server_key_iv: "c6365692b50103d3be34ae4530cb9d5f"
server_key_version: "encryptionkey@0"
I20260927 05:10:58.649067 19182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:58.674283 19182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:58.675421 19182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:58.675829 19182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:58.697214 19182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39333
I20260927 05:10:58.697256 19247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39333 every 8 connection(s)
I20260927 05:10:58.698932 19182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:10:58.703650 19248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:10:58.704774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 19182
I20260927 05:10:58.704972 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:10:58.705443 388 external_mini_cluster.cc:1428] Setting key 1aef2b3f764b09ee8e8a5363e4c59f49
I20260927 05:10:58.721007 19248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e: Bootstrap starting.
I20260927 05:10:58.724223 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:58.707678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.0.97.62:39333)
0927 05:10:58.709079 (+ 1401us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:58.709098 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:58.709165 (+ 67us) server_negotiation.cc:407] Connection header received
0927 05:10:58.711453 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:58.711492 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:58.712314 (+ 822us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:58.712867 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:58.713491 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:58.715550 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:58.717328 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:58.718660 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:58.718874 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:58.720532 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:58.720547 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:58.720575 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:58.721462 (+ 887us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:58.721591 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:58.721633 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:58.721806 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:58.722129 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 05:10:58.722538 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":456,"threads_started":1}
I20260927 05:10:58.725929 19248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e: Neither blocks nor log segments found. Creating new log.
I20260927 05:10:58.727140 19248 log.cc:824] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:10:58.730340 19248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e: No bootstrap required, opened a new log
I20260927 05:10:58.738664 19248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354dab88871a4f22a75b0785d9f9299e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } }
I20260927 05:10:58.739173 19248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:10:58.739280 19248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354dab88871a4f22a75b0785d9f9299e, State: Initialized, Role: FOLLOWER
I20260927 05:10:58.740085 19248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354dab88871a4f22a75b0785d9f9299e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } }
I20260927 05:10:58.740409 19248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:10:58.740583 19248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:10:58.740815 19248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:10:58.743494 19248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354dab88871a4f22a75b0785d9f9299e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } }
I20260927 05:10:58.744225 19248 leader_election.cc:302] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 354dab88871a4f22a75b0785d9f9299e; no voters:
I20260927 05:10:58.744903 19248 leader_election.cc:288] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:10:58.745152 19253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:10:58.746245 19253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [term 1 LEADER]: Becoming Leader. State: Replica: 354dab88871a4f22a75b0785d9f9299e, State: Running, Role: LEADER
I20260927 05:10:58.747025 19253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354dab88871a4f22a75b0785d9f9299e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } }
I20260927 05:10:58.747846 19248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:10:58.750653 19255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 354dab88871a4f22a75b0785d9f9299e. Latest consensus state: current_term: 1 leader_uuid: "354dab88871a4f22a75b0785d9f9299e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354dab88871a4f22a75b0785d9f9299e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } }
I20260927 05:10:58.751320 19255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:58.751111 19254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "354dab88871a4f22a75b0785d9f9299e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "354dab88871a4f22a75b0785d9f9299e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } }
I20260927 05:10:58.751509 19254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e [sys.catalog]: This master's current role is: LEADER
I20260927 05:10:58.754271 19262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:10:58.759317 19262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:10:58.769981 19262 catalog_manager.cc:1347] Generated new cluster ID: ef7153849bb24ef9a3a493a0f8a365d7
I20260927 05:10:58.770298 19262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:10:58.794962 19262 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:10:58.796646 19262 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:10:58.806483 19262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e: Generated new TSK 0
I20260927 05:10:58.807384 19262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:10:58.816789 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--builtin_ntp_servers=127.0.97.20:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:10:59.215775 19272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:59.216150 19272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:59.216251 19272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:59.225275 19272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:59.225481 19272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:59.225680 19272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:10:59.237632 19272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:59.239486 19272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:59.242069 19272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:59.258750 19277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:59.258823 19280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:59.258736 19278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:59.259637 19272 server_base.cc:1034] running on GCE node
I20260927 05:10:59.260425 19272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:59.261584 19272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:59.262877 19272 hybrid_clock.cc:648] HybridClock initialized: now 1790485859262814 us; error 56 us; skew 500 ppm
I20260927 05:10:59.263273 19272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:10:59.266090 19272 webserver.cc:460] Webserver started at http://127.0.97.1:35093/ using document root <none> and password file <none>
I20260927 05:10:59.266903 19272 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:59.267051 19272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:59.267441 19272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:59.270651 19272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "c049a639a9144241aad076377ed837a6"
format_stamp: "Formatted at 2026-09-27 05:10:59 on dist-test-slave-hgs0"
server_key: "0f1e95f9ac2ee5e7b3065b311735a488"
server_key_iv: "1c4d80c47396d556655bcef23c723927"
server_key_version: "encryptionkey@0"
I20260927 05:10:59.271575 19272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c049a639a9144241aad076377ed837a6"
format_stamp: "Formatted at 2026-09-27 05:10:59 on dist-test-slave-hgs0"
server_key: "0f1e95f9ac2ee5e7b3065b311735a488"
server_key_iv: "1c4d80c47396d556655bcef23c723927"
server_key_version: "encryptionkey@0"
I20260927 05:10:59.278057 19272 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:10:59.282239 19286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:59.283895 19272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:10:59.284219 19272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "c049a639a9144241aad076377ed837a6"
format_stamp: "Formatted at 2026-09-27 05:10:59 on dist-test-slave-hgs0"
server_key: "0f1e95f9ac2ee5e7b3065b311735a488"
server_key_iv: "1c4d80c47396d556655bcef23c723927"
server_key_version: "encryptionkey@0"
I20260927 05:10:59.284523 19272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:59.299058 19272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:59.300271 19272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:59.300611 19272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:59.301815 19272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:59.303546 19272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:59.303683 19272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:59.303843 19272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:59.303941 19272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:59.342094 19272 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42269
I20260927 05:10:59.342139 19398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42269 every 8 connection(s)
I20260927 05:10:59.343657 19272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:10:59.352708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 19272
I20260927 05:10:59.352931 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:10:59.353434 388 external_mini_cluster.cc:1428] Setting key 2534bfd38604cfcd992c711b3d1f8ea2
I20260927 05:10:59.356493 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--builtin_ntp_servers=127.0.97.20:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:10:59.358623 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:59.346557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35481 (local address 127.0.97.62:39333)
0927 05:10:59.347068 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:59.347073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:59.348552 (+ 1479us) server_negotiation.cc:407] Connection header received
0927 05:10:59.350642 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:59.350648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:59.350745 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:59.350879 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:59.352209 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:59.353243 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:59.355296 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:59.355752 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:59.355911 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:59.357629 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:59.357635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:59.357639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:59.357660 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:59.357706 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:59.357719 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:59.357828 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:59.358121 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 05:10:59.358242 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":275,"threads_started":1}
I20260927 05:10:59.359826 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:59.347263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39333 (local address 127.0.97.1:35481)
0927 05:10:59.348350 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:59.348396 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:59.350323 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:59.351147 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:59.351157 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:59.351269 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:59.352009 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:59.352039 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:59.353408 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:59.353414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:59.355119 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:59.355134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:59.356066 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:59.356074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:59.356348 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:59.357225 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:10:59.357228 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:59.357282 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:10:59.357380 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:59.357827 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:59.357843 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:59.357855 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 05:10:59.358088 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 05:10:59.358579 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":322,"threads_started":1}
I20260927 05:10:59.361869 19399 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39333
I20260927 05:10:59.362264 19399 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:59.363150 19399 heartbeater.cc:511] Master 127.0.97.62:39333 requested a full tablet report, sending...
I20260927 05:10:59.365582 19213 ts_manager.cc:194] Registered new tserver with Master: c049a639a9144241aad076377ed837a6 (127.0.97.1:42269)
I20260927 05:10:59.367924 19213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35481
W20260927 05:10:59.718709 19403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:10:59.719017 19403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:10:59.719105 19403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:10:59.726770 19403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:10:59.726895 19403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:10:59.727008 19403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:10:59.737299 19403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:10:59.738731 19403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:10:59.741101 19403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:10:59.757177 19408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:59.757177 19409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:10:59.757272 19411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:10:59.758602 19403 server_base.cc:1034] running on GCE node
I20260927 05:10:59.759275 19403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:10:59.760382 19403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:10:59.761583 19403 hybrid_clock.cc:648] HybridClock initialized: now 1790485859761517 us; error 65 us; skew 500 ppm
I20260927 05:10:59.761890 19403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:10:59.764418 19403 webserver.cc:460] Webserver started at http://127.0.97.2:35357/ using document root <none> and password file <none>
I20260927 05:10:59.765100 19403 fs_manager.cc:359] Metadata directory not provided
I20260927 05:10:59.765216 19403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:10:59.765599 19403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:10:59.768661 19403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "6fc56780536c4e709c6d9992444852c9"
format_stamp: "Formatted at 2026-09-27 05:10:59 on dist-test-slave-hgs0"
server_key: "dbbd427dbdaf82941b56037c30ba5e7b"
server_key_iv: "1ace58a3a0181dff72a6320c87d9d6c7"
server_key_version: "encryptionkey@0"
I20260927 05:10:59.769475 19403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6fc56780536c4e709c6d9992444852c9"
format_stamp: "Formatted at 2026-09-27 05:10:59 on dist-test-slave-hgs0"
server_key: "dbbd427dbdaf82941b56037c30ba5e7b"
server_key_iv: "1ace58a3a0181dff72a6320c87d9d6c7"
server_key_version: "encryptionkey@0"
I20260927 05:10:59.774916 19403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 05:10:59.778630 19417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:59.780267 19403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:10:59.780471 19403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "6fc56780536c4e709c6d9992444852c9"
format_stamp: "Formatted at 2026-09-27 05:10:59 on dist-test-slave-hgs0"
server_key: "dbbd427dbdaf82941b56037c30ba5e7b"
server_key_iv: "1ace58a3a0181dff72a6320c87d9d6c7"
server_key_version: "encryptionkey@0"
I20260927 05:10:59.780717 19403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:10:59.802795 19403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:10:59.803782 19403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:10:59.804176 19403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:10:59.805325 19403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:10:59.807071 19403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:10:59.807219 19403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:59.807353 19403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:10:59.807440 19403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:10:59.844165 19403 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41495
I20260927 05:10:59.844257 19529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41495 every 8 connection(s)
I20260927 05:10:59.845579 19403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:10:59.847276 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 19403
I20260927 05:10:59.847503 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:10:59.848196 388 external_mini_cluster.cc:1428] Setting key f19768579785a8be317c29561a907451
I20260927 05:10:59.851730 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--builtin_ntp_servers=127.0.97.20:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:10:59.861521 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:59.848973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46225 (local address 127.0.97.62:39333)
0927 05:10:59.849273 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 05:10:59.849285 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:10:59.851298 (+ 2013us) server_negotiation.cc:407] Connection header received
0927 05:10:59.853691 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:10:59.853697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:10:59.853806 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:10:59.853951 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:10:59.855249 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:59.856176 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:59.858011 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:10:59.858467 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:10:59.858642 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:59.860562 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:10:59.860572 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:10:59.860579 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:10:59.860612 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:10:59.860665 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:10:59.860688 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:10:59.860808 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 05:10:59.861147 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 05:10:59.861246 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 05:10:59.863148 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:10:59.849945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39333 (local address 127.0.97.2:46225)
0927 05:10:59.851122 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0927 05:10:59.851174 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 05:10:59.853326 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:10:59.854241 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:10:59.854252 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:10:59.854368 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:10:59.855048 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:59.855068 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:59.856381 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:59.856386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:59.857805 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:10:59.857820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:10:59.858752 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:10:59.858760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:10:59.859079 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:10:59.860024 (+ 945us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:10:59.860029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:10:59.860095 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:10:59.860247 (+ 152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:10:59.860839 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:10:59.860853 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:10:59.860865 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 05:10:59.861087 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 05:10:59.861580 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":391,"threads_started":1}
I20260927 05:10:59.864854 19530 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39333
I20260927 05:10:59.865265 19530 heartbeater.cc:464] Registering TS with master...
I20260927 05:10:59.866109 19530 heartbeater.cc:511] Master 127.0.97.62:39333 requested a full tablet report, sending...
I20260927 05:10:59.867713 19213 ts_manager.cc:194] Registered new tserver with Master: 6fc56780536c4e709c6d9992444852c9 (127.0.97.2:41495)
I20260927 05:10:59.868640 19213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:46225
W20260927 05:11:00.229801 19533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:00.230137 19533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:00.230283 19533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:00.237911 19533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:00.238046 19533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:00.238222 19533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:11:00.248611 19533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:00.250239 19533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:00.252466 19533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:00.267038 19538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:00.267123 19539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:00.267079 19541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:00.268312 19533 server_base.cc:1034] running on GCE node
I20260927 05:11:00.269026 19533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:00.270133 19533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:00.271442 19533 hybrid_clock.cc:648] HybridClock initialized: now 1790485860271381 us; error 54 us; skew 500 ppm
I20260927 05:11:00.271848 19533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:11:00.274659 19533 webserver.cc:460] Webserver started at http://127.0.97.3:35939/ using document root <none> and password file <none>
I20260927 05:11:00.275503 19533 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:00.275664 19533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:00.276144 19533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:00.279685 19533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2"
format_stamp: "Formatted at 2026-09-27 05:11:00 on dist-test-slave-hgs0"
server_key: "736625a6395687abfb840a3338635593"
server_key_iv: "fb65bd3b3db2d3abea10f1ccf682620b"
server_key_version: "encryptionkey@0"
I20260927 05:11:00.280742 19533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2"
format_stamp: "Formatted at 2026-09-27 05:11:00 on dist-test-slave-hgs0"
server_key: "736625a6395687abfb840a3338635593"
server_key_iv: "fb65bd3b3db2d3abea10f1ccf682620b"
server_key_version: "encryptionkey@0"
I20260927 05:11:00.287194 19533 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260927 05:11:00.291625 19547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:00.293365 19533 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:11:00.293576 19533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2"
format_stamp: "Formatted at 2026-09-27 05:11:00 on dist-test-slave-hgs0"
server_key: "736625a6395687abfb840a3338635593"
server_key_iv: "fb65bd3b3db2d3abea10f1ccf682620b"
server_key_version: "encryptionkey@0"
I20260927 05:11:00.293841 19533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:00.325450 19533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:00.326508 19533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:00.326776 19533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:00.327865 19533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:11:00.329806 19533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:11:00.329901 19533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:00.329985 19533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:11:00.330044 19533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:00.366838 19533 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42845
I20260927 05:11:00.367035 19659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42845 every 8 connection(s)
I20260927 05:11:00.368377 19533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:11:00.371915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 19533
I20260927 05:11:00.372162 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:11:00.372156 19399 heartbeater.cc:503] Master 127.0.97.62:39333 was elected leader, sending a full tablet report...
I20260927 05:11:00.372833 388 external_mini_cluster.cc:1428] Setting key 594c0f8c137cad81d1ae201912497fb9
I20260927 05:11:00.382756 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.371257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57089 (local address 127.0.97.62:39333)
0927 05:11:00.371834 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:00.371842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:00.373533 (+ 1691us) server_negotiation.cc:407] Connection header received
0927 05:11:00.376052 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:00.376057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:00.376123 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:00.376217 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:00.377381 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.378139 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.379761 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.380300 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.380408 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.381897 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:00.381902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:00.381906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:00.381929 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:00.381980 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:00.381996 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:00.382081 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:00.382306 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 05:11:00.382400 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":296,"threads_started":1}
I20260927 05:11:00.383841 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.372017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39333 (local address 127.0.97.3:57089)
0927 05:11:00.373350 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:00.373409 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:00.375670 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:00.376447 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:00.376457 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:00.376562 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:00.377224 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.377247 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.378294 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.378298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.379600 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.379612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.380510 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.380515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.380735 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.381508 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:11:00.381510 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:00.381570 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:11:00.381658 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:00.382102 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:00.382110 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:00.382116 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:11:00.382278 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 05:11:00.382635 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":350,"threads_started":1}
I20260927 05:11:00.385615 19660 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39333
I20260927 05:11:00.386011 19660 heartbeater.cc:464] Registering TS with master...
I20260927 05:11:00.387125 19660 heartbeater.cc:511] Master 127.0.97.62:39333 requested a full tablet report, sending...
I20260927 05:11:00.389005 19212 ts_manager.cc:194] Registered new tserver with Master: ef474771ea1c488eb6ae5b7faf4fb0f2 (127.0.97.3:42845)
I20260927 05:11:00.390043 19212 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:57089
I20260927 05:11:00.401847 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:11:00.413533 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.406294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.0.97.62:39333)
0927 05:11:00.406584 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:00.406592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:00.406892 (+ 300us) server_negotiation.cc:407] Connection header received
0927 05:11:00.407045 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:00.407050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:00.407119 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:00.407237 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:00.408011 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.408831 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.410619 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.411066 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.411183 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.412543 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:00.412549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:00.412552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:00.412573 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:00.412610 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:00.412623 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:00.412760 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:00.413068 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 05:11:00.413175 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 05:11:00.418493 19212 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:11:00.425683 19212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:11:00.468181 19675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.440139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42269 (local address 127.0.0.1:37416)
0927 05:11:00.441569 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:00.441638 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:00.441931 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:00.457924 (+ 15993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:00.457941 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:00.458027 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:00.458876 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.458902 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.461413 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.461423 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.464409 (+ 2986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.464427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.467088 (+ 2661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.467094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.467329 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.467389 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 05:11:00.467539 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 05:11:00.467786 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"thread_start_us":19104,"threads_started":1}
I20260927 05:11:00.469072 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.440379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.0.97.1:42269)
0927 05:11:00.456821 (+ 16442us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:00.456836 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:00.456892 (+ 56us) server_negotiation.cc:407] Connection header received
0927 05:11:00.457045 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:00.457060 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:00.457454 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:00.457816 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:00.459291 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.461248 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.464685 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.466344 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.466495 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.466841 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:00.468150 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0927 05:11:00.468664 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16121,"thread_start_us":17722,"threads_started":1}
I20260927 05:11:00.470578 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.443296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42845 (local address 127.0.0.1:60106)
0927 05:11:00.459963 (+ 16667us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:00.460003 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:00.460139 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:00.461807 (+ 1668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:00.461812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:00.461849 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:00.462379 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.462392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.464334 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.464343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.466861 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.466880 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.469667 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.469672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.469829 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.469861 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 05:11:00.469977 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 05:11:00.470186 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16485,"mutex_wait_us":155,"thread_start_us":16272,"threads_started":1}
I20260927 05:11:00.470584 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.443706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.0.97.3:42845)
0927 05:11:00.461076 (+ 17370us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:00.461086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:00.461134 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:11:00.461232 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:00.461240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:00.461580 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:00.461828 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:00.462665 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.464191 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.467700 (+ 3509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.469064 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.469289 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.469540 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:00.470030 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 05:11:00.470216 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17154,"thread_start_us":16939,"threads_started":1}
I20260927 05:11:00.470791 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.441290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.0.97.2:41495)
0927 05:11:00.459027 (+ 17737us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:00.459042 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:00.459842 (+ 800us) server_negotiation.cc:407] Connection header received
0927 05:11:00.460261 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:00.460272 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:00.461650 (+ 1378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:00.461922 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:00.462983 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.464757 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.467265 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.468963 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.469112 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.469414 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:00.470175 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 05:11:00.470380 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17529,"thread_start_us":17400,"threads_started":1}
I20260927 05:11:00.471421 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.440578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41495 (local address 127.0.0.1:52110)
0927 05:11:00.459688 (+ 19110us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:00.459712 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:00.460036 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:00.462217 (+ 2181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:00.462223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:00.462268 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:00.462758 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.462772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.464919 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.464926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.466864 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.466880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.469656 (+ 2776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.469664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.469875 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.469909 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 05:11:00.470006 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 05:11:00.471141 (+ 1135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18914,"mutex_wait_us":989,"thread_start_us":1570,"threads_started":1}
I20260927 05:11:00.474114 19334 tablet_service.cc:1467] Processing CreateTablet for tablet e4ed67fd24dd4e9e94985ff06468d522 (DEFAULT_TABLE table=test-table [id=a6956ff0e57d424d9bcac04442854f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:11:00.475219 19465 tablet_service.cc:1467] Processing CreateTablet for tablet e4ed67fd24dd4e9e94985ff06468d522 (DEFAULT_TABLE table=test-table [id=a6956ff0e57d424d9bcac04442854f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:11:00.476013 19334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ed67fd24dd4e9e94985ff06468d522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:00.475605 19595 tablet_service.cc:1467] Processing CreateTablet for tablet e4ed67fd24dd4e9e94985ff06468d522 (DEFAULT_TABLE table=test-table [id=a6956ff0e57d424d9bcac04442854f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:11:00.476708 19465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ed67fd24dd4e9e94985ff06468d522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:00.477200 19595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ed67fd24dd4e9e94985ff06468d522. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:00.490868 19679 tablet_bootstrap.cc:492] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: Bootstrap starting.
I20260927 05:11:00.496551 19679 tablet_bootstrap.cc:654] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:00.496899 19681 tablet_bootstrap.cc:492] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: Bootstrap starting.
I20260927 05:11:00.498438 19680 tablet_bootstrap.cc:492] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: Bootstrap starting.
I20260927 05:11:00.498413 19679 log.cc:824] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:00.502607 19681 tablet_bootstrap.cc:654] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:00.502744 19680 tablet_bootstrap.cc:654] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:00.504106 19681 log.cc:824] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:00.504280 19680 log.cc:824] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:00.504581 19679 tablet_bootstrap.cc:492] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: No bootstrap required, opened a new log
I20260927 05:11:00.504977 19679 ts_tablet_manager.cc:1397] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260927 05:11:00.507773 19680 tablet_bootstrap.cc:492] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: No bootstrap required, opened a new log
I20260927 05:11:00.507870 19681 tablet_bootstrap.cc:492] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: No bootstrap required, opened a new log
I20260927 05:11:00.508184 19680 ts_tablet_manager.cc:1397] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 05:11:00.508241 19681 ts_tablet_manager.cc:1397] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260927 05:11:00.517449 19680 raft_consensus.cc:348] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 } } peers { permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 } } peers { permanent_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42845 } }
I20260927 05:11:00.518020 19680 raft_consensus.cc:374] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:00.518151 19680 raft_consensus.cc:729] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c049a639a9144241aad076377ed837a6, State: Initialized, Role: FOLLOWER
I20260927 05:11:00.518059 19679 raft_consensus.cc:348] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 } } peers { permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 } } peers { permanent_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42845 } }
I20260927 05:11:00.518600 19679 raft_consensus.cc:374] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:00.518707 19679 raft_consensus.cc:729] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc56780536c4e709c6d9992444852c9, State: Initialized, Role: FOLLOWER
I20260927 05:11:00.518978 19680 consensus_queue.cc:260] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 } } peers { permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 } } peers { permanent_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42845 } }
I20260927 05:11:00.519431 19679 consensus_queue.cc:260] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 } } peers { permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 } } peers { permanent_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42845 } }
I20260927 05:11:00.521441 19680 ts_tablet_manager.cc:1428] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 05:11:00.521230 19681 raft_consensus.cc:348] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 } } peers { permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 } } peers { permanent_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42845 } }
I20260927 05:11:00.521875 19681 raft_consensus.cc:374] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:00.521999 19681 raft_consensus.cc:729] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef474771ea1c488eb6ae5b7faf4fb0f2, State: Initialized, Role: FOLLOWER
I20260927 05:11:00.522809 19530 heartbeater.cc:503] Master 127.0.97.62:39333 was elected leader, sending a full tablet report...
I20260927 05:11:00.522909 19681 consensus_queue.cc:260] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 } } peers { permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 } } peers { permanent_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42845 } }
I20260927 05:11:00.524153 19660 heartbeater.cc:503] Master 127.0.97.62:39333 was elected leader, sending a full tablet report...
I20260927 05:11:00.526459 19679 ts_tablet_manager.cc:1428] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: Time spent starting tablet: real 0.021s user 0.009s sys 0.008s
I20260927 05:11:00.527915 19681 ts_tablet_manager.cc:1428] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: Time spent starting tablet: real 0.020s user 0.014s sys 0.002s
W20260927 05:11:00.596454 19400 tablet.cc:2369] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:11:00.597749 19531 tablet.cc:2369] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:11:00.620364 19661 tablet.cc:2369] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:11:00.676478 19687 raft_consensus.cc:482] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:11:00.676874 19687 raft_consensus.cc:504] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 } } peers { permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 } } peers { permanent_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42845 } }
I20260927 05:11:00.678692 19687 leader_election.cc:288] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c049a639a9144241aad076377ed837a6 (127.0.97.1:42269), 6fc56780536c4e709c6d9992444852c9 (127.0.97.2:41495)
I20260927 05:11:00.685456 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.679365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45877 (local address 127.0.97.2:41495)
0927 05:11:00.679600 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:00.679607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:00.679629 (+ 22us) server_negotiation.cc:407] Connection header received
0927 05:11:00.679716 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:00.679722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:00.679823 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:00.679916 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:00.680553 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.681520 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.683508 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.684597 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.684696 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.684765 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:00.685095 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 05:11:00.685259 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 05:11:00.685762 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.679154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41495 (local address 127.0.97.3:45877)
0927 05:11:00.679354 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:00.679382 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:00.679522 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:00.680062 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:00.680067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:00.680096 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:00.680377 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.680392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.681719 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.681725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.683326 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.683349 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.684740 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.684746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.684922 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.684959 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:11:00.685054 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 05:11:00.685223 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 05:11:00.686355 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.679713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36809 (local address 127.0.97.1:42269)
0927 05:11:00.679906 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:00.679912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:00.680247 (+ 335us) server_negotiation.cc:407] Connection header received
0927 05:11:00.680425 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:00.680430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:00.680533 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:00.680615 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:00.681221 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.682091 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.684203 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.685430 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.685556 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.685667 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:00.685982 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 05:11:00.686126 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:11:00.686463 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.679211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42269 (local address 127.0.97.3:36809)
0927 05:11:00.680134 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:00.680151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:00.680291 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:00.680672 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:00.680677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:00.680701 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:00.681054 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.681065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.682274 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.682278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.683999 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:00.684012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.685596 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.685613 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:00.685796 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.685842 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 05:11:00.685944 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 05:11:00.686071 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":5,"spinlock_wait_cycles":20864,"thread_start_us":532,"threads_started":1}
I20260927 05:11:00.686924 19485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed67fd24dd4e9e94985ff06468d522" candidate_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc56780536c4e709c6d9992444852c9" is_pre_election: true
I20260927 05:11:00.687569 19485 raft_consensus.cc:2454] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef474771ea1c488eb6ae5b7faf4fb0f2 in term 0.
I20260927 05:11:00.687579 19354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed67fd24dd4e9e94985ff06468d522" candidate_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c049a639a9144241aad076377ed837a6" is_pre_election: true
I20260927 05:11:00.688185 19354 raft_consensus.cc:2454] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef474771ea1c488eb6ae5b7faf4fb0f2 in term 0.
I20260927 05:11:00.688354 19550 leader_election.cc:302] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fc56780536c4e709c6d9992444852c9, ef474771ea1c488eb6ae5b7faf4fb0f2; no voters:
I20260927 05:11:00.688812 19687 raft_consensus.cc:2781] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:11:00.688975 19687 raft_consensus.cc:482] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:11:00.689085 19687 raft_consensus.cc:3037] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:00.691246 19687 raft_consensus.cc:504] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 } } peers { permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 } } peers { permanent_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42845 } }
I20260927 05:11:00.691911 19687 leader_election.cc:288] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [CANDIDATE]: Term 1 election: Requested vote from peers c049a639a9144241aad076377ed837a6 (127.0.97.1:42269), 6fc56780536c4e709c6d9992444852c9 (127.0.97.2:41495)
I20260927 05:11:00.692361 19354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed67fd24dd4e9e94985ff06468d522" candidate_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c049a639a9144241aad076377ed837a6"
I20260927 05:11:00.692488 19485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed67fd24dd4e9e94985ff06468d522" candidate_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc56780536c4e709c6d9992444852c9"
I20260927 05:11:00.692569 19354 raft_consensus.cc:3037] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:00.692768 19485 raft_consensus.cc:3037] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:00.694903 19354 raft_consensus.cc:2454] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef474771ea1c488eb6ae5b7faf4fb0f2 in term 1.
I20260927 05:11:00.695005 19485 raft_consensus.cc:2454] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef474771ea1c488eb6ae5b7faf4fb0f2 in term 1.
I20260927 05:11:00.695364 19549 leader_election.cc:302] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c049a639a9144241aad076377ed837a6, ef474771ea1c488eb6ae5b7faf4fb0f2; no voters:
I20260927 05:11:00.695680 19687 raft_consensus.cc:2781] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:00.696259 19687 raft_consensus.cc:686] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 1 LEADER]: Becoming Leader. State: Replica: ef474771ea1c488eb6ae5b7faf4fb0f2, State: Running, Role: LEADER
I20260927 05:11:00.696827 19687 consensus_queue.cc:237] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 } } peers { permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 } } peers { permanent_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42845 } }
I20260927 05:11:00.704617 19212 catalog_manager.cc:5495] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef474771ea1c488eb6ae5b7faf4fb0f2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef474771ea1c488eb6ae5b7faf4fb0f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42845 } health_report { overall_health: HEALTHY } } }
I20260927 05:11:00.775070 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.769301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.0.97.3:42845)
0927 05:11:00.769493 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:00.769498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:00.769638 (+ 140us) server_negotiation.cc:407] Connection header received
0927 05:11:00.769830 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:00.769834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:00.769925 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:00.770065 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:00.770437 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.771183 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.772739 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.773166 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.773261 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.773827 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:00.774379 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:00.774533 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:00.774811 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 05:11:00.774863 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:11:00.784516 19485 raft_consensus.cc:1261] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9 [term 1 FOLLOWER]: Refusing update from remote peer ef474771ea1c488eb6ae5b7faf4fb0f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:11:00.784516 19354 raft_consensus.cc:1261] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6 [term 1 FOLLOWER]: Refusing update from remote peer ef474771ea1c488eb6ae5b7faf4fb0f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:11:00.785885 19689 consensus_queue.cc:1035] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c049a639a9144241aad076377ed837a6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:11:00.786386 19687 consensus_queue.cc:1035] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc56780536c4e709c6d9992444852c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:11:00.800173 19692 mvcc.cc:205] Tried to move back new op lower bound from 7333830085759848448 to 7333830085418536960. Current Snapshot: MvccSnapshot[applied={T|T < 7333830085759848448}]
I20260927 05:11:00.818527 19213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:11:00.818826 19213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:11:00.823294 19213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 354dab88871a4f22a75b0785d9f9299e: Sending DeleteTablet for 3 replicas of tablet e4ed67fd24dd4e9e94985ff06468d522
I20260927 05:11:00.824808 19334 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ed67fd24dd4e9e94985ff06468d522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:11:00 UTC) from {username='slave'} at 127.0.0.1:37416
I20260927 05:11:00.824970 19465 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ed67fd24dd4e9e94985ff06468d522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:11:00 UTC) from {username='slave'} at 127.0.0.1:52110
I20260927 05:11:00.826094 19595 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ed67fd24dd4e9e94985ff06468d522 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:11:00 UTC) from {username='slave'} at 127.0.0.1:60106
I20260927 05:11:00.826598 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 19272
I20260927 05:11:00.830830 19703 tablet_replica.cc:330] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: stopping tablet replica
I20260927 05:11:00.831563 19703 raft_consensus.cc:2229] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:11:00.832577 19703 raft_consensus.cc:2258] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:11:00.833580 19701 tablet_replica.cc:330] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: stopping tablet replica
I20260927 05:11:00.833690 19702 tablet_replica.cc:330] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: stopping tablet replica
I20260927 05:11:00.834143 19701 raft_consensus.cc:2229] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:11:00.834298 19702 raft_consensus.cc:2229] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:11:00.834771 19701 raft_consensus.cc:2258] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:11:00.834949 19702 raft_consensus.cc:2258] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:11:00.837612 19701 ts_tablet_manager.cc:1905] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:11:00.837613 19703 ts_tablet_manager.cc:1905] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:11:00.845561 19701 ts_tablet_manager.cc:1918] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:11:00.845738 19701 log.cc:1196] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-0/wal/wals/e4ed67fd24dd4e9e94985ff06468d522
I20260927 05:11:00.846050 19703 ts_tablet_manager.cc:1918] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:11:00.846256 19703 log.cc:1196] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/e4ed67fd24dd4e9e94985ff06468d522
I20260927 05:11:00.846310 19701 ts_tablet_manager.cc:1939] T e4ed67fd24dd4e9e94985ff06468d522 P c049a639a9144241aad076377ed837a6: Deleting consensus metadata
I20260927 05:11:00.846808 19703 ts_tablet_manager.cc:1939] T e4ed67fd24dd4e9e94985ff06468d522 P ef474771ea1c488eb6ae5b7faf4fb0f2: Deleting consensus metadata
I20260927 05:11:00.848806 19198 catalog_manager.cc:4841] TS c049a639a9144241aad076377ed837a6 (127.0.97.1:42269): tablet e4ed67fd24dd4e9e94985ff06468d522 (table test-table [id=a6956ff0e57d424d9bcac04442854f76]) successfully deleted
I20260927 05:11:00.850263 19702 ts_tablet_manager.cc:1905] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:11:00.851850 19199 catalog_manager.cc:4841] TS ef474771ea1c488eb6ae5b7faf4fb0f2 (127.0.97.3:42845): tablet e4ed67fd24dd4e9e94985ff06468d522 (table test-table [id=a6956ff0e57d424d9bcac04442854f76]) successfully deleted
I20260927 05:11:00.853613 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.828926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.0.97.1:42269)
0927 05:11:00.829878 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:00.829889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:00.835745 (+ 5856us) server_negotiation.cc:407] Connection header received
0927 05:11:00.836142 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:00.836148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:00.836266 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:00.836386 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:00.837059 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.838172 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.840304 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.840972 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.841073 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.842799 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:00.842813 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:00.842841 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:00.843793 (+ 952us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:00.843869 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:00.843897 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:00.844016 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:00.852948 (+ 8932us) server_negotiation.cc:299] Negotiation successful
0927 05:11:00.853133 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757}
I20260927 05:11:00.855211 19394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:00.860538 19702 ts_tablet_manager.cc:1918] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:11:00.860860 19702 log.cc:1196] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/e4ed67fd24dd4e9e94985ff06468d522
I20260927 05:11:00.861791 19702 ts_tablet_manager.cc:1939] T e4ed67fd24dd4e9e94985ff06468d522 P 6fc56780536c4e709c6d9992444852c9: Deleting consensus metadata
I20260927 05:11:00.864674 19199 catalog_manager.cc:4841] TS 6fc56780536c4e709c6d9992444852c9 (127.0.97.2:41495): tablet e4ed67fd24dd4e9e94985ff06468d522 (table test-table [id=a6956ff0e57d424d9bcac04442854f76]) successfully deleted
I20260927 05:11:00.962219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 19272
I20260927 05:11:00.980027 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 19403
I20260927 05:11:00.988154 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:00.980968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.0.97.2:41495)
0927 05:11:00.981151 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:00.981155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:00.981175 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:11:00.981236 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:00.981240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:00.981334 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:00.981426 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:00.981897 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.982676 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.984736 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:00.985267 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:00.985385 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:00.986592 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:00.986601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:00.986618 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:00.987291 (+ 673us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:00.987349 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:00.987369 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:00.987456 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:00.987638 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 05:11:00.987734 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:11:00.988914 19525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:01.085924 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 19403
I20260927 05:11:01.110590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 19533
I20260927 05:11:01.125789 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:01.111575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.0.97.3:42845)
0927 05:11:01.111790 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:01.111799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:01.111873 (+ 74us) server_negotiation.cc:407] Connection header received
0927 05:11:01.112124 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:01.112128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:01.112236 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:01.112338 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:01.112899 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:01.113794 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:01.115900 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:01.116624 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:01.116752 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:01.124142 (+ 7390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:01.124153 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:01.124170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:01.124873 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:01.124934 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:01.124962 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:01.125079 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:01.125298 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 05:11:01.125396 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:11:01.126554 19655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:01.256718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 19533
I20260927 05:11:01.278918 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 19182
I20260927 05:11:01.280330 19243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:01.398675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 19182
2026-09-27T05:11:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3255 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:01 dist-test-slave-hgs0 krb5kdc[19711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:01 dist-test-slave-hgs0 krb5kdc[19711](info): set up 2 sockets
Sep 27 05:11:01 dist-test-slave-hgs0 krb5kdc[19711](info): commencing operation
krb5kdc: starting...
W20260927 05:11:02.596638 388 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:02 dist-test-slave-hgs0 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:11:02.868105 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40167
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:03.239271 19724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:03.239617 19724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:03.239779 19724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:03.247730 19724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:03.247865 19724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:03.247972 19724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:03.248064 19724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:03.248128 19724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:03.258435 19724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:03.260313 19724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:03.261849 19724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:03.274575 19729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:03.274626 19732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:03.274590 19730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:03.275759 19724 server_base.cc:1034] running on GCE node
I20260927 05:11:03.276769 19724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:03.278442 19724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:03.279712 19724 hybrid_clock.cc:648] HybridClock initialized: now 1790485863279661 us; error 43 us; skew 500 ppm
I20260927 05:11:03.280222 19724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:03 dist-test-slave-hgs0 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:03.283672 19724 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:11:03.285662 19724 webserver.cc:460] Webserver started at http://127.0.97.62:33829/ using document root <none> and password file <none>
I20260927 05:11:03.286454 19724 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:03.286614 19724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:03.286981 19724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:03.290280 19724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9"
format_stamp: "Formatted at 2026-09-27 05:11:03 on dist-test-slave-hgs0"
server_key: "251dd8afbc4ad702d6352c7ba822f6b9"
server_key_iv: "a49fdd46a181836277e6b3cea4bd09b1"
server_key_version: "encryptionkey@0"
I20260927 05:11:03.291126 19724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9"
format_stamp: "Formatted at 2026-09-27 05:11:03 on dist-test-slave-hgs0"
server_key: "251dd8afbc4ad702d6352c7ba822f6b9"
server_key_iv: "a49fdd46a181836277e6b3cea4bd09b1"
server_key_version: "encryptionkey@0"
I20260927 05:11:03.296965 19724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:11:03.300891 19739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:03.302289 19724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:11:03.302485 19724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9"
format_stamp: "Formatted at 2026-09-27 05:11:03 on dist-test-slave-hgs0"
server_key: "251dd8afbc4ad702d6352c7ba822f6b9"
server_key_iv: "a49fdd46a181836277e6b3cea4bd09b1"
server_key_version: "encryptionkey@0"
I20260927 05:11:03.302714 19724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:03.332755 19724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:03.337479 19724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:03.337808 19724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:03.359212 19724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40167
I20260927 05:11:03.359236 19790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40167 every 8 connection(s)
I20260927 05:11:03.360891 19724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:03.365013 19791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:03.366528 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 19724
I20260927 05:11:03.366746 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:03.367305 388 external_mini_cluster.cc:1428] Setting key 0f37f2859660fd28fc1f06518208dc93
I20260927 05:11:03.383694 19791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9: Bootstrap starting.
Sep 27 05:11:03 dist-test-slave-hgs0 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:03.388430 19791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:03.389801 19791 log.cc:824] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:03.393563 19791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9: No bootstrap required, opened a new log
I20260927 05:11:03.397271 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:03.369622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.0.97.62:40167)
0927 05:11:03.371234 (+ 1612us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:03.371274 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:03.371329 (+ 55us) server_negotiation.cc:407] Connection header received
0927 05:11:03.374312 (+ 2983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:03.374351 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:03.375088 (+ 737us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:03.375685 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:03.378156 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:03.380328 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:03.381941 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:03.382751 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:03.383011 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:03.388564 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:03.388595 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:03.388623 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:03.388695 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:03.392469 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:03.393462 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:03.393470 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:03.393477 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:03.393703 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:03.394172 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:03.394176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:03.394178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:03.394757 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:03.394998 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:03.395554 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 05:11:03.396123 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":763,"threads_started":1}
I20260927 05:11:03.402351 19791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40167 } }
I20260927 05:11:03.402812 19791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:03.402930 19791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9ab37dc1da41b7b2e44835cbd1e4c9, State: Initialized, Role: FOLLOWER
I20260927 05:11:03.403543 19791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40167 } }
I20260927 05:11:03.403820 19791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:03.403991 19791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:03.404181 19791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:03.406755 19791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40167 } }
I20260927 05:11:03.407392 19791 leader_election.cc:302] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a9ab37dc1da41b7b2e44835cbd1e4c9; no voters:
I20260927 05:11:03.408156 19791 leader_election.cc:288] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:03.408301 19796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:03.409044 19796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [term 1 LEADER]: Becoming Leader. State: Replica: 5a9ab37dc1da41b7b2e44835cbd1e4c9, State: Running, Role: LEADER
I20260927 05:11:03.409652 19796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40167 } }
I20260927 05:11:03.410481 19791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:03.412835 19797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40167 } } }
I20260927 05:11:03.413216 19797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:03.415026 19801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:03.414912 19798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a9ab37dc1da41b7b2e44835cbd1e4c9. Latest consensus state: current_term: 1 leader_uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9ab37dc1da41b7b2e44835cbd1e4c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40167 } } }
I20260927 05:11:03.415253 19798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:03.423467 19801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:03.435176 19801 catalog_manager.cc:1347] Generated new cluster ID: 2fe6c1bf724d440fb7035ebb3158c390
I20260927 05:11:03.435321 19801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:03.459195 19801 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:03.461345 19801 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:03.472671 19801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9: Generated new TSK 0
I20260927 05:11:03.473696 19801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260927 05:11:03.549736 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40167
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:03.948728 19817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:03.949030 19817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:03.949127 19817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:03.956575 19817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:03.956701 19817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:03.956816 19817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:11:03.967010 19817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:03.968344 19817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:03.969676 19817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:03.983655 19825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:03.983613 19823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:03.983713 19822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:03.984675 19817 server_base.cc:1034] running on GCE node
I20260927 05:11:03.985337 19817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:03.986426 19817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:03.987705 19817 hybrid_clock.cc:648] HybridClock initialized: now 1790485863987628 us; error 58 us; skew 500 ppm
I20260927 05:11:03.988235 19817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:03 dist-test-slave-hgs0 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:03.991717 19817 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260927 05:11:03.993530 19817 webserver.cc:460] Webserver started at http://127.0.97.1:43425/ using document root <none> and password file <none>
I20260927 05:11:03.994313 19817 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:03.994470 19817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:03.994807 19817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:03.998235 19817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "dd945860a49347dea83e8af5c62b8eb1"
format_stamp: "Formatted at 2026-09-27 05:11:03 on dist-test-slave-hgs0"
server_key: "eee36db560ccfe5b2f1196ded9e6baa5"
server_key_iv: "79c11d6e2de9b69d413e1fb451f6c7dd"
server_key_version: "encryptionkey@0"
I20260927 05:11:03.999212 19817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dd945860a49347dea83e8af5c62b8eb1"
format_stamp: "Formatted at 2026-09-27 05:11:03 on dist-test-slave-hgs0"
server_key: "eee36db560ccfe5b2f1196ded9e6baa5"
server_key_iv: "79c11d6e2de9b69d413e1fb451f6c7dd"
server_key_version: "encryptionkey@0"
I20260927 05:11:04.005743 19817 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 05:11:04.009869 19832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:04.011476 19817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 05:11:04.011658 19817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "dd945860a49347dea83e8af5c62b8eb1"
format_stamp: "Formatted at 2026-09-27 05:11:03 on dist-test-slave-hgs0"
server_key: "eee36db560ccfe5b2f1196ded9e6baa5"
server_key_iv: "79c11d6e2de9b69d413e1fb451f6c7dd"
server_key_version: "encryptionkey@0"
I20260927 05:11:04.011878 19817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:04.038265 19817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:04.042964 19817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:04.043299 19817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:04.044418 19817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:11:04.046205 19817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:11:04.046319 19817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:04.046464 19817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:11:04.046540 19817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:04.082161 19817 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44993
I20260927 05:11:04.082242 19944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44993 every 8 connection(s)
I20260927 05:11:04.083534 19817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:11:04.092504 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 19817
I20260927 05:11:04.092715 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:11:04.093243 388 external_mini_cluster.cc:1428] Setting key c4c9479f4ae6d471053bbcf4f3cc908f
Sep 27 05:11:04 dist-test-slave-hgs0 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:04.105186 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:04.086300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36833 (local address 127.0.97.62:40167)
0927 05:11:04.086823 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:04.086830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:04.087763 (+ 933us) server_negotiation.cc:407] Connection header received
0927 05:11:04.089731 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:04.089735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:04.089811 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:04.089912 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:04.091520 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:04.092541 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:04.094364 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:04.094814 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:04.094966 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:04.098998 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:04.099004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:04.099007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:04.099043 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:04.101776 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:04.102607 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:04.102614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:04.102630 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:04.102754 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:04.103222 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:04.103228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:04.103233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:04.103527 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:04.103705 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:04.104547 (+ 842us) server_negotiation.cc:299] Negotiation successful
0927 05:11:04.104804 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":233,"threads_started":1}
I20260927 05:11:04.106282 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:04.086840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40167 (local address 127.0.97.1:36833)
0927 05:11:04.087574 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:04.087634 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:04.089439 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:04.090139 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:04.090148 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:04.090740 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:04.091345 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:04.091364 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:04.092742 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:04.092750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:04.094195 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:04.094209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:04.095066 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:04.095075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:04.095341 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:04.096133 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:04.096158 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:04.098721 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:04.101948 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:04.101958 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:04.101979 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:04.102467 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:04.102876 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:04.102881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:04.102884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:04.103089 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:04.103685 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:04.103694 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:04.104138 (+ 444us) client_negotiation.cc:769] Sending connection context
0927 05:11:04.104441 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 05:11:04.105043 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":277,"threads_started":1}
I20260927 05:11:04.108110 19945 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40167
I20260927 05:11:04.108456 19945 heartbeater.cc:464] Registering TS with master...
I20260927 05:11:04.109232 19945 heartbeater.cc:511] Master 127.0.97.62:40167 requested a full tablet report, sending...
I20260927 05:11:04.112036 19756 ts_manager.cc:194] Registered new tserver with Master: dd945860a49347dea83e8af5c62b8eb1 (127.0.97.1:44993)
I20260927 05:11:04.114483 19756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36833
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260927 05:11:04.160331 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40167
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:04.526530 19951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:04.526830 19951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:04.526925 19951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:04.534338 19951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:04.534471 19951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:04.534584 19951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:11:04.544603 19951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:04.545890 19951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:04.547211 19951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:04.561657 19956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:04.561609 19957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:04.561576 19959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:04.562481 19951 server_base.cc:1034] running on GCE node
I20260927 05:11:04.563226 19951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:04.564244 19951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:04.565487 19951 hybrid_clock.cc:648] HybridClock initialized: now 1790485864565426 us; error 45 us; skew 500 ppm
I20260927 05:11:04.565848 19951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:04 dist-test-slave-hgs0 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:04.569416 19951 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260927 05:11:04.571038 19951 webserver.cc:460] Webserver started at http://127.0.97.2:40663/ using document root <none> and password file <none>
I20260927 05:11:04.571754 19951 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:04.571859 19951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:04.572237 19951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:04.575433 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "ea7c4a65ce8b4b24845cd9e12955a13f"
format_stamp: "Formatted at 2026-09-27 05:11:04 on dist-test-slave-hgs0"
server_key: "ee41add209e7f38c86590f9e34b663ff"
server_key_iv: "1c554ef9aa0ad07a165baf9f6506cd4b"
server_key_version: "encryptionkey@0"
I20260927 05:11:04.576371 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ea7c4a65ce8b4b24845cd9e12955a13f"
format_stamp: "Formatted at 2026-09-27 05:11:04 on dist-test-slave-hgs0"
server_key: "ee41add209e7f38c86590f9e34b663ff"
server_key_iv: "1c554ef9aa0ad07a165baf9f6506cd4b"
server_key_version: "encryptionkey@0"
I20260927 05:11:04.581974 19951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 05:11:04.585758 19966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:04.587219 19951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:11:04.587405 19951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "ea7c4a65ce8b4b24845cd9e12955a13f"
format_stamp: "Formatted at 2026-09-27 05:11:04 on dist-test-slave-hgs0"
server_key: "ee41add209e7f38c86590f9e34b663ff"
server_key_iv: "1c554ef9aa0ad07a165baf9f6506cd4b"
server_key_version: "encryptionkey@0"
I20260927 05:11:04.587630 19951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:04.602816 19951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:04.608053 19951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:04.608361 19951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:04.609439 19951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:11:04.610939 19951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:11:04.611049 19951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:04.611196 19951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:11:04.611271 19951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:04.647541 19951 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33893
I20260927 05:11:04.647583 20078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33893 every 8 connection(s)
I20260927 05:11:04.648854 19951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:11:04.650660 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 19951
I20260927 05:11:04.650825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:11:04.651312 388 external_mini_cluster.cc:1428] Setting key c46b87f823cdd9a6ac7325b41e9c49d5
Sep 27 05:11:04 dist-test-slave-hgs0 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:04.672293 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:04.651652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37013 (local address 127.0.97.62:40167)
0927 05:11:04.652157 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:04.652163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:04.653438 (+ 1275us) server_negotiation.cc:407] Connection header received
0927 05:11:04.655319 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:04.655327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:04.655427 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:04.655549 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:04.657180 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:04.658310 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:04.659988 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:04.660764 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:04.660917 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:04.665298 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:04.665304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:04.665308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:04.665346 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:04.668246 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:04.669368 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:04.669373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:04.669376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:04.669553 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:04.670119 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:04.670123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:04.670125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:04.670426 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:04.670580 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:04.671659 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0927 05:11:04.671926 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":263,"threads_started":1}
I20260927 05:11:04.673846 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:04.652285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40167 (local address 127.0.97.2:37013)
0927 05:11:04.653254 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:04.653325 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:04.655026 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:04.655762 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:04.655771 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:04.656403 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:04.656997 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:04.657013 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:04.658479 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:04.658483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:04.659806 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:04.659818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:04.661002 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:04.661008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:04.661215 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:04.662053 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:04.662077 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:04.664987 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:04.668443 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:04.668456 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:04.668480 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:04.669187 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:04.669695 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:04.669702 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:04.669709 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:04.669993 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:04.670603 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:04.670615 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:04.671127 (+ 512us) client_negotiation.cc:769] Sending connection context
0927 05:11:04.671475 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 05:11:04.672175 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":303,"threads_started":1}
I20260927 05:11:04.675854 20079 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40167
I20260927 05:11:04.676280 20079 heartbeater.cc:464] Registering TS with master...
I20260927 05:11:04.677162 20079 heartbeater.cc:511] Master 127.0.97.62:40167 requested a full tablet report, sending...
I20260927 05:11:04.678988 19756 ts_manager.cc:194] Registered new tserver with Master: ea7c4a65ce8b4b24845cd9e12955a13f (127.0.97.2:33893)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260927 05:11:04.680104 19756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260927 05:11:04.715843 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40167
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:05.079427 20085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:05.079741 20085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:05.079842 20085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:05.087759 20085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:05.087888 20085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:05.088155 20085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:11:05.099032 20085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:05.100471 20085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:05.101862 20085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:05.117081 20091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:05.117187 20093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:05.117223 20090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:05.118615 20085 server_base.cc:1034] running on GCE node
I20260927 05:11:05.118994 19945 heartbeater.cc:503] Master 127.0.97.62:40167 was elected leader, sending a full tablet report...
I20260927 05:11:05.119647 20085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:05.121331 20085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:05.122594 20085 hybrid_clock.cc:648] HybridClock initialized: now 1790485865122541 us; error 59 us; skew 500 ppm
I20260927 05:11:05.123025 20085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:05 dist-test-slave-hgs0 krb5kdc[19711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:05.126869 20085 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260927 05:11:05.128510 20085 webserver.cc:460] Webserver started at http://127.0.97.3:35155/ using document root <none> and password file <none>
I20260927 05:11:05.129240 20085 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:05.129343 20085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:05.129617 20085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:05.132810 20085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "3d72c7b4b74a4097971056ba681f64d5"
format_stamp: "Formatted at 2026-09-27 05:11:05 on dist-test-slave-hgs0"
server_key: "95298559d69fd62a43a26f1781ac18d5"
server_key_iv: "0e26e858332ee27ef26e05ce3aa54fd2"
server_key_version: "encryptionkey@0"
I20260927 05:11:05.133641 20085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3d72c7b4b74a4097971056ba681f64d5"
format_stamp: "Formatted at 2026-09-27 05:11:05 on dist-test-slave-hgs0"
server_key: "95298559d69fd62a43a26f1781ac18d5"
server_key_iv: "0e26e858332ee27ef26e05ce3aa54fd2"
server_key_version: "encryptionkey@0"
I20260927 05:11:05.140162 20085 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 05:11:05.144541 20100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:05.146247 20085 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 05:11:05.146461 20085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "3d72c7b4b74a4097971056ba681f64d5"
format_stamp: "Formatted at 2026-09-27 05:11:05 on dist-test-slave-hgs0"
server_key: "95298559d69fd62a43a26f1781ac18d5"
server_key_iv: "0e26e858332ee27ef26e05ce3aa54fd2"
server_key_version: "encryptionkey@0"
I20260927 05:11:05.146735 20085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:05.199437 20085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:05.204635 20085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:05.204960 20085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:05.206074 20085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:11:05.207823 20085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:11:05.207906 20085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:05.208125 20085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:11:05.208179 20085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:05.245289 20085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34951
I20260927 05:11:05.245345 20212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34951 every 8 connection(s)
I20260927 05:11:05.246701 20085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:11:05.254952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 20085
I20260927 05:11:05.255156 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:11:05.255587 388 external_mini_cluster.cc:1428] Setting key bf03af73fcb5fc006988453dab8632ff
Sep 27 05:11:05 dist-test-slave-hgs0 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:05.269737 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.249497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52737 (local address 127.0.97.62:40167)
0927 05:11:05.250021 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:05.250026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:05.251004 (+ 978us) server_negotiation.cc:407] Connection header received
0927 05:11:05.253157 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:05.253163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:05.253273 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:05.253403 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:05.255098 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.256105 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.258307 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.258938 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.259080 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.262892 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:05.262897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:05.262901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:05.262938 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:05.266050 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:05.266931 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:05.266938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:05.266941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:05.267128 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:05.267655 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:05.267660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:05.267662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:05.268088 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:05.268217 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:05.269084 (+ 867us) server_negotiation.cc:299] Negotiation successful
0927 05:11:05.269364 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":261,"threads_started":1}
I20260927 05:11:05.270885 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.249995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40167 (local address 127.0.97.3:52737)
0927 05:11:05.250786 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.250832 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:05.252840 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:05.253688 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:05.253698 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:05.254275 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:05.254887 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.254908 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.256301 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.256308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.258134 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.258148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.259187 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.259192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.259396 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.260174 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:05.260209 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:05.262632 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:05.266235 (+ 3603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:05.266244 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:05.266260 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:05.266771 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:05.267296 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:05.267300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:05.267303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:05.267525 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:05.268226 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:05.268234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:05.268632 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 05:11:05.268920 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 05:11:05.269413 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":344,"threads_started":1}
I20260927 05:11:05.272444 20213 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40167
I20260927 05:11:05.272740 20213 heartbeater.cc:464] Registering TS with master...
I20260927 05:11:05.273640 20213 heartbeater.cc:511] Master 127.0.97.62:40167 requested a full tablet report, sending...
I20260927 05:11:05.275247 19756 ts_manager.cc:194] Registered new tserver with Master: 3d72c7b4b74a4097971056ba681f64d5 (127.0.97.3:34951)
I20260927 05:11:05.276158 19756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52737
I20260927 05:11:05.284323 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:11:05.302732 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.288847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.0.97.62:40167)
0927 05:11:05.289048 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:05.289053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:05.289578 (+ 525us) server_negotiation.cc:407] Connection header received
0927 05:11:05.289739 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:05.289743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:05.289813 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:05.289910 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:05.291435 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.292552 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.294004 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.294452 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.294579 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.297316 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:05.297322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:05.297325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:05.297367 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:05.300150 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:05.300849 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:05.300853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:05.300856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:05.300985 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:05.301369 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:05.301372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:05.301374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:05.301728 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:05.301914 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:05.302295 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 05:11:05.302488 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 05:11:05.307202 19756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:11:05.314694 19756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:11:05.341243 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.328276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.0.97.1:44993)
0927 05:11:05.329165 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:05.329179 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:05.329852 (+ 673us) server_negotiation.cc:407] Connection header received
0927 05:11:05.330273 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:05.330282 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:05.330563 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:05.330742 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:05.331843 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.333722 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.337210 (+ 3488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.339093 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.339250 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.339587 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:05.340576 (+ 989us) server_negotiation.cc:299] Negotiation successful
0927 05:11:05.340839 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":443,"threads_started":1}
I20260927 05:11:05.342412 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.328070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44993 (local address 127.0.0.1:60250)
0927 05:11:05.329712 (+ 1642us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.329756 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:05.330000 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:05.331071 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:05.331078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:05.331146 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:05.331589 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.331602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.333892 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.333896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.337005 (+ 3109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.337021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.339833 (+ 2812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.339844 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.340214 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.340292 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 05:11:05.341673 (+ 1381us) client_negotiation.cc:240] Negotiation successful
0927 05:11:05.341903 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1383,"mutex_wait_us":1,"spinlock_wait_cycles":6528,"threads_started":1}
I20260927 05:11:05.343248 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.329108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.0.97.3:34951)
0927 05:11:05.329902 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:05.329914 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:05.330995 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 05:11:05.331348 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:05.331360 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:05.331646 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:05.331843 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:05.333138 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.335277 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.338751 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.340994 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.341135 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.341436 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:05.342560 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0927 05:11:05.342784 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":1745,"threads_started":1}
I20260927 05:11:05.343835 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.328447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34951 (local address 127.0.0.1:46442)
0927 05:11:05.330844 (+ 2397us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.330867 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:05.331128 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:05.332314 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:05.332322 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:05.332360 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:05.332883 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.332897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.335524 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.335532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.338500 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.338522 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.341665 (+ 3143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.341676 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.341975 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.342029 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 05:11:05.342145 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 05:11:05.342356 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2239,"thread_start_us":899,"threads_started":1}
I20260927 05:11:05.347862 19880 tablet_service.cc:1467] Processing CreateTablet for tablet f57c19787a164b72bc459a768703999c (DEFAULT_TABLE table=test-table [id=32f73745e0d94267b185a711da7fb2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:11:05.347878 20148 tablet_service.cc:1467] Processing CreateTablet for tablet f57c19787a164b72bc459a768703999c (DEFAULT_TABLE table=test-table [id=32f73745e0d94267b185a711da7fb2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:11:05.349376 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57c19787a164b72bc459a768703999c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:05.349429 19880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57c19787a164b72bc459a768703999c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:05.351578 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.330716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.0.97.2:33893)
0927 05:11:05.343676 (+ 12960us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:05.343685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:05.343722 (+ 37us) server_negotiation.cc:407] Connection header received
0927 05:11:05.343849 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:05.343857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:05.344228 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:05.344444 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:05.344849 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.346698 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.348891 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.350450 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.350591 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.350859 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:05.351067 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 05:11:05.351232 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12818,"thread_start_us":12703,"threads_started":1}
I20260927 05:11:05.351590 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.330249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33893 (local address 127.0.0.1:41284)
0927 05:11:05.342816 (+ 12567us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.342842 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:05.342983 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:05.344424 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:05.344429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:05.344458 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:05.344669 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.344682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.346875 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.346880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.348705 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.348731 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.350625 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.350631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.350823 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.350887 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 05:11:05.350995 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 05:11:05.351142 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12422,"thread_start_us":865,"threads_started":1}
I20260927 05:11:05.355561 20014 tablet_service.cc:1467] Processing CreateTablet for tablet f57c19787a164b72bc459a768703999c (DEFAULT_TABLE table=test-table [id=32f73745e0d94267b185a711da7fb2ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:11:05.357105 20014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57c19787a164b72bc459a768703999c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:05.372993 20232 tablet_bootstrap.cc:492] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: Bootstrap starting.
I20260927 05:11:05.376754 20233 tablet_bootstrap.cc:492] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1: Bootstrap starting.
I20260927 05:11:05.377815 20234 tablet_bootstrap.cc:492] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: Bootstrap starting.
I20260927 05:11:05.378409 20232 tablet_bootstrap.cc:654] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:05.379824 20232 log.cc:824] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:05.381536 20234 tablet_bootstrap.cc:654] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:05.381702 20233 tablet_bootstrap.cc:654] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:05.382938 20233 log.cc:824] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:05.383064 20234 log.cc:824] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:05.390406 20232 tablet_bootstrap.cc:492] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: No bootstrap required, opened a new log
I20260927 05:11:05.390775 20232 ts_tablet_manager.cc:1397] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20260927 05:11:05.391705 20234 tablet_bootstrap.cc:492] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: No bootstrap required, opened a new log
I20260927 05:11:05.391757 20233 tablet_bootstrap.cc:492] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1: No bootstrap required, opened a new log
I20260927 05:11:05.392047 20234 ts_tablet_manager.cc:1397] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.004s
I20260927 05:11:05.392074 20233 ts_tablet_manager.cc:1397] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260927 05:11:05.400379 20234 raft_consensus.cc:348] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 } } peers { permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 } } peers { permanent_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33893 } }
I20260927 05:11:05.400975 20234 raft_consensus.cc:374] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:05.401089 20234 raft_consensus.cc:729] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea7c4a65ce8b4b24845cd9e12955a13f, State: Initialized, Role: FOLLOWER
I20260927 05:11:05.401896 20234 consensus_queue.cc:260] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 } } peers { permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 } } peers { permanent_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33893 } }
I20260927 05:11:05.402704 20232 raft_consensus.cc:348] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 } } peers { permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 } } peers { permanent_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33893 } }
I20260927 05:11:05.403249 20232 raft_consensus.cc:374] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:05.403388 20232 raft_consensus.cc:729] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d72c7b4b74a4097971056ba681f64d5, State: Initialized, Role: FOLLOWER
I20260927 05:11:05.403614 20233 raft_consensus.cc:348] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 } } peers { permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 } } peers { permanent_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33893 } }
I20260927 05:11:05.404191 20233 raft_consensus.cc:374] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:05.404322 20233 raft_consensus.cc:729] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd945860a49347dea83e8af5c62b8eb1, State: Initialized, Role: FOLLOWER
I20260927 05:11:05.404325 20079 heartbeater.cc:503] Master 127.0.97.62:40167 was elected leader, sending a full tablet report...
I20260927 05:11:05.405162 20234 ts_tablet_manager.cc:1428] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260927 05:11:05.405089 20233 consensus_queue.cc:260] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 } } peers { permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 } } peers { permanent_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33893 } }
I20260927 05:11:05.405812 20232 consensus_queue.cc:260] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 } } peers { permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 } } peers { permanent_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33893 } }
I20260927 05:11:05.407423 20213 heartbeater.cc:503] Master 127.0.97.62:40167 was elected leader, sending a full tablet report...
I20260927 05:11:05.407692 20233 ts_tablet_manager.cc:1428] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260927 05:11:05.408354 20232 ts_tablet_manager.cc:1428] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
W20260927 05:11:05.498581 20214 tablet.cc:2369] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:11:05.584475 20238 raft_consensus.cc:482] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:11:05.584772 20238 raft_consensus.cc:504] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 } } peers { permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 } } peers { permanent_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33893 } }
I20260927 05:11:05.586519 20238 leader_election.cc:288] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd945860a49347dea83e8af5c62b8eb1 (127.0.97.1:44993), 3d72c7b4b74a4097971056ba681f64d5 (127.0.97.3:34951)
W20260927 05:11:05.586540 19946 tablet.cc:2369] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:11:05.594242 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.586996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33993 (local address 127.0.97.3:34951)
0927 05:11:05.587361 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:05.587372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:05.588368 (+ 996us) server_negotiation.cc:407] Connection header received
0927 05:11:05.588558 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:05.588567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:05.588656 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:05.588779 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:05.589436 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.590226 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.592264 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.593102 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.593205 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.593348 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:05.593798 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 05:11:05.593935 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260927 05:11:05.594409 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.586819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44993 (local address 127.0.97.2:49367)
0927 05:11:05.587681 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.587707 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:05.587908 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:05.588395 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:05.588402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:05.588434 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:05.588847 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.588860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.590111 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.590116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.592039 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.592055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.593394 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.593408 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.593572 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.593612 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 05:11:05.593690 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 05:11:05.593865 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":80,"thread_start_us":737,"threads_started":1}
I20260927 05:11:05.594547 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.587703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49367 (local address 127.0.97.1:44993)
0927 05:11:05.587987 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:05.587994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:05.588050 (+ 56us) server_negotiation.cc:407] Connection header received
0927 05:11:05.588138 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:05.588144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:05.588230 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:05.588350 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:05.589042 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.589952 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.592211 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.593123 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.593299 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.593411 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:05.593755 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 05:11:05.593905 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:11:05.594841 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.586933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34951 (local address 127.0.97.2:33993)
0927 05:11:05.588254 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.588276 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:05.588409 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:05.588830 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:05.588834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:05.588855 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:05.589278 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.589292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.590393 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.590397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.592092 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:05.592105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.593352 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.593359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:05.593587 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.593634 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 05:11:05.593746 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 05:11:05.593905 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"mutex_wait_us":120,"thread_start_us":995,"threads_started":1}
I20260927 05:11:05.595707 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57c19787a164b72bc459a768703999c" candidate_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd945860a49347dea83e8af5c62b8eb1" is_pre_election: true
I20260927 05:11:05.596095 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57c19787a164b72bc459a768703999c" candidate_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d72c7b4b74a4097971056ba681f64d5" is_pre_election: true
I20260927 05:11:05.596290 19900 raft_consensus.cc:2454] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea7c4a65ce8b4b24845cd9e12955a13f in term 0.
I20260927 05:11:05.596619 20168 raft_consensus.cc:2454] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea7c4a65ce8b4b24845cd9e12955a13f in term 0.
I20260927 05:11:05.597115 19969 leader_election.cc:302] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd945860a49347dea83e8af5c62b8eb1, ea7c4a65ce8b4b24845cd9e12955a13f; no voters:
I20260927 05:11:05.597659 20238 raft_consensus.cc:2781] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:11:05.597816 20238 raft_consensus.cc:482] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:11:05.597929 20238 raft_consensus.cc:3037] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:05.600557 20238 raft_consensus.cc:504] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 } } peers { permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 } } peers { permanent_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33893 } }
I20260927 05:11:05.601401 20238 leader_election.cc:288] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [CANDIDATE]: Term 1 election: Requested vote from peers dd945860a49347dea83e8af5c62b8eb1 (127.0.97.1:44993), 3d72c7b4b74a4097971056ba681f64d5 (127.0.97.3:34951)
I20260927 05:11:05.602011 19900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57c19787a164b72bc459a768703999c" candidate_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd945860a49347dea83e8af5c62b8eb1"
I20260927 05:11:05.602114 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57c19787a164b72bc459a768703999c" candidate_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d72c7b4b74a4097971056ba681f64d5"
I20260927 05:11:05.602285 19900 raft_consensus.cc:3037] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:05.602429 20168 raft_consensus.cc:3037] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:05.605346 19900 raft_consensus.cc:2454] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea7c4a65ce8b4b24845cd9e12955a13f in term 1.
I20260927 05:11:05.605419 20168 raft_consensus.cc:2454] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea7c4a65ce8b4b24845cd9e12955a13f in term 1.
I20260927 05:11:05.606062 19969 leader_election.cc:302] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd945860a49347dea83e8af5c62b8eb1, ea7c4a65ce8b4b24845cd9e12955a13f; no voters:
I20260927 05:11:05.606537 20238 raft_consensus.cc:2781] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:05.607296 20238 raft_consensus.cc:686] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 1 LEADER]: Becoming Leader. State: Replica: ea7c4a65ce8b4b24845cd9e12955a13f, State: Running, Role: LEADER
I20260927 05:11:05.608091 20238 consensus_queue.cc:237] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 } } peers { permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 } } peers { permanent_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33893 } }
I20260927 05:11:05.613683 19756 catalog_manager.cc:5495] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f reported cstate change: term changed from 0 to 1, leader changed from <none> to ea7c4a65ce8b4b24845cd9e12955a13f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea7c4a65ce8b4b24845cd9e12955a13f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33893 } health_report { overall_health: HEALTHY } } }
I20260927 05:11:05.644407 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.638007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.0.97.2:33893)
0927 05:11:05.638251 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:05.638256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:05.638276 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:11:05.638502 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:05.638507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:05.638597 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:05.638736 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:05.639161 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.640138 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.641729 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.642250 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.642374 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.642920 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:05.643650 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:05.643823 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:05.643877 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 05:11:05.644067 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 05:11:05.651100 20080 tablet.cc:2369] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:11:05.654054 19900 raft_consensus.cc:1261] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1 [term 1 FOLLOWER]: Refusing update from remote peer ea7c4a65ce8b4b24845cd9e12955a13f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:11:05.654095 20168 raft_consensus.cc:1261] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5 [term 1 FOLLOWER]: Refusing update from remote peer ea7c4a65ce8b4b24845cd9e12955a13f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:11:05.655257 20243 consensus_queue.cc:1035] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d72c7b4b74a4097971056ba681f64d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:11:05.655706 20238 consensus_queue.cc:1035] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd945860a49347dea83e8af5c62b8eb1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:11:05.673362 20248 mvcc.cc:205] Tried to move back new op lower bound from 7333830105706987520 to 7333830105536221184. Current Snapshot: MvccSnapshot[applied={T|T < 7333830105706987520}]
I20260927 05:11:05.674854 20246 mvcc.cc:205] Tried to move back new op lower bound from 7333830105706987520 to 7333830105536221184. Current Snapshot: MvccSnapshot[applied={T|T < 7333830105706987520}]
I20260927 05:11:05.680683 20247 mvcc.cc:205] Tried to move back new op lower bound from 7333830105706987520 to 7333830105536221184. Current Snapshot: MvccSnapshot[applied={T|T < 7333830105706987520}]
I20260927 05:11:05.690232 19755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:11:05.690606 19755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:11:05.694857 19755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a9ab37dc1da41b7b2e44835cbd1e4c9: Sending DeleteTablet for 3 replicas of tablet f57c19787a164b72bc459a768703999c
I20260927 05:11:05.696218 19880 tablet_service.cc:1514] Processing DeleteTablet for tablet f57c19787a164b72bc459a768703999c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:11:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60250
I20260927 05:11:05.696573 20148 tablet_service.cc:1514] Processing DeleteTablet for tablet f57c19787a164b72bc459a768703999c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:11:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46442
I20260927 05:11:05.697009 20014 tablet_service.cc:1514] Processing DeleteTablet for tablet f57c19787a164b72bc459a768703999c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:11:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41284
I20260927 05:11:05.697247 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 19817
I20260927 05:11:05.697618 20256 tablet_replica.cc:330] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: stopping tablet replica
I20260927 05:11:05.698159 20256 raft_consensus.cc:2229] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:11:05.698792 20256 raft_consensus.cc:2258] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:11:05.701186 20256 ts_tablet_manager.cc:1905] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:11:05.709489 20256 ts_tablet_manager.cc:1918] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:11:05.709784 20256 log.cc:1196] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/f57c19787a164b72bc459a768703999c
I20260927 05:11:05.710601 20256 ts_tablet_manager.cc:1939] T f57c19787a164b72bc459a768703999c P 3d72c7b4b74a4097971056ba681f64d5: Deleting consensus metadata
I20260927 05:11:05.713164 19743 catalog_manager.cc:4841] TS 3d72c7b4b74a4097971056ba681f64d5 (127.0.97.3:34951): tablet f57c19787a164b72bc459a768703999c (table test-table [id=32f73745e0d94267b185a711da7fb2ed]) successfully deleted
Sep 27 05:11:05 dist-test-slave-hgs0 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260927 05:11:05.718794 20257 tablet_replica.cc:330] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: stopping tablet replica
I20260927 05:11:05.719393 20257 raft_consensus.cc:2229] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:11:05.720292 20257 raft_consensus.cc:2258] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:11:05.721172 20255 tablet_replica.cc:330] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1: stopping tablet replica
I20260927 05:11:05.721647 20255 raft_consensus.cc:2229] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:11:05.722199 20255 raft_consensus.cc:2258] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:11:05.722913 20257 ts_tablet_manager.cc:1905] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:11:05.724334 20255 ts_tablet_manager.cc:1905] T f57c19787a164b72bc459a768703999c P dd945860a49347dea83e8af5c62b8eb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:11:05.725353 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.698424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.0.97.1:44993)
0927 05:11:05.698611 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:05.698618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:05.704558 (+ 5940us) server_negotiation.cc:407] Connection header received
0927 05:11:05.704751 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:05.704757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:05.704849 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:05.704955 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:05.706849 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.708090 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.710012 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.710706 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.710870 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.716629 (+ 5759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:05.716652 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:05.716679 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:05.716727 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:05.720769 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:05.722168 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:05.722180 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:05.722190 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:05.722444 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:05.723075 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:05.723080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:05.723083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:05.723599 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:05.723906 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:05.724693 (+ 787us) server_negotiation.cc:299] Negotiation successful
0927 05:11:05.724961 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 05:11:05.727023 19940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:05.731001 20257 ts_tablet_manager.cc:1918] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:11:05.731236 20257 log.cc:1196] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/f57c19787a164b72bc459a768703999c
I20260927 05:11:05.731841 20257 ts_tablet_manager.cc:1939] T f57c19787a164b72bc459a768703999c P ea7c4a65ce8b4b24845cd9e12955a13f: Deleting consensus metadata
I20260927 05:11:05.734418 19743 catalog_manager.cc:4841] TS ea7c4a65ce8b4b24845cd9e12955a13f (127.0.97.2:33893): tablet f57c19787a164b72bc459a768703999c (table test-table [id=32f73745e0d94267b185a711da7fb2ed]) successfully deleted
I20260927 05:11:05.829558 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 19817
W20260927 05:11:05.848788 19742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44993 (error 108)
I20260927 05:11:05.848948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 19951
I20260927 05:11:05.850714 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.850112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44993 (local address 127.0.0.1:60278)
0927 05:11:05.850350 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.850508 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 05:11:05.850979 19742 catalog_manager.cc:4584] TS dd945860a49347dea83e8af5c62b8eb1 (127.0.97.1:44993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f57c19787a164b72bc459a768703999c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44993: connect: Connection refused (error 111)
Sep 27 05:11:05 dist-test-slave-hgs0 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260927 05:11:05.867244 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.850296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.0.97.2:33893)
0927 05:11:05.850526 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:05.850539 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:05.850560 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:11:05.850616 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:05.850621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:05.850692 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:05.850812 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:05.852362 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.853211 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.854534 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:05.854986 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:05.855149 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:05.859924 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:05.859996 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:05.860023 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:05.860098 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:05.863524 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:05.864618 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:05.864627 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:05.864630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:05.864843 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:05.865339 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:05.865344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:05.865346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:05.865816 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:05.866231 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:05.866539 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 05:11:05.866807 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:11:05.868196 20074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:05.904663 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.904204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44993 (local address 127.0.0.1:60288)
0927 05:11:05.904407 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.904516 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 05:11:05.905825 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.905417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44993 (local address 127.0.0.1:60302)
0927 05:11:05.905573 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.905673 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 05:11:05.966671 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.966147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44993 (local address 127.0.0.1:60316)
0927 05:11:05.966396 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.966509 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 05:11:05.967823 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:05.967475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44993 (local address 127.0.0.1:60326)
0927 05:11:05.967611 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:05.967683 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 05:11:05.990542 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 19951
I20260927 05:11:06.011852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 20085
Sep 27 05:11:06 dist-test-slave-hgs0 krb5kdc[19711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260927 05:11:06.029268 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:06.012861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.0.97.3:34951)
0927 05:11:06.013095 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:06.013100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:06.013121 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:11:06.013266 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:06.013270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:06.013356 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:06.013455 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:06.014957 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:06.015762 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:06.017291 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:06.017722 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:06.017827 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:06.022572 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:06.022589 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:06.022606 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:06.022648 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:06.026014 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:06.026845 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:06.026854 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:06.026856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:06.027016 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:06.027426 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:06.027432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:06.027435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:06.028001 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:06.028317 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:06.028648 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 05:11:06.028887 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 05:11:06.030097 20208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:06.048906 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:06.048410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44993 (local address 127.0.0.1:60338)
0927 05:11:06.048648 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:06.048757 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 05:11:06.050153 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:06.049658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44993 (local address 127.0.0.1:60344)
0927 05:11:06.049882 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:06.049986 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 05:11:06.127964 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 20085
I20260927 05:11:06.146584 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 19724
I20260927 05:11:06.148027 19786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:06.266413 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:06.265743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44993 (local address 127.0.0.1:60358)
0927 05:11:06.266009 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:06.266204 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 05:11:06.266752 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 19724
2026-09-27T05:11:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4867 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T05:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:06Z Disabled control of system clock
I20260927 05:11:06.349905 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44255
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:11:06.711573 20267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:06.711915 20267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:06.712081 20267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:06.720181 20267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:06.720306 20267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:06.720350 20267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:06.720402 20267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:06.720448 20267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:06.730189 20267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:06.731524 20267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:06.733386 20267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:06.745867 20273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:06.746074 20275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:06.745965 20272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:06.747211 20267 server_base.cc:1034] running on GCE node
I20260927 05:11:06.748198 20267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:06.749787 20267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:06.751071 20267 hybrid_clock.cc:648] HybridClock initialized: now 1790485866751004 us; error 59 us; skew 500 ppm
I20260927 05:11:06.751500 20267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:11:06.763640 20267 webserver.cc:460] Webserver started at http://127.0.97.62:45273/ using document root <none> and password file <none>
I20260927 05:11:06.764631 20267 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:06.764804 20267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:06.765193 20267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:06.768749 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "ef56fda272b747e9a59ee1c83eb77328"
format_stamp: "Formatted at 2026-09-27 05:11:06 on dist-test-slave-hgs0"
server_key: "872858c0ea21133f3a84d5a8b1a06cee"
server_key_iv: "cafa39e23453980caf05f23afbc4259b"
server_key_version: "encryptionkey@0"
I20260927 05:11:06.769840 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef56fda272b747e9a59ee1c83eb77328"
format_stamp: "Formatted at 2026-09-27 05:11:06 on dist-test-slave-hgs0"
server_key: "872858c0ea21133f3a84d5a8b1a06cee"
server_key_iv: "cafa39e23453980caf05f23afbc4259b"
server_key_version: "encryptionkey@0"
I20260927 05:11:06.776180 20267 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 05:11:06.780182 20281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:06.781844 20267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:11:06.782086 20267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "ef56fda272b747e9a59ee1c83eb77328"
format_stamp: "Formatted at 2026-09-27 05:11:06 on dist-test-slave-hgs0"
server_key: "872858c0ea21133f3a84d5a8b1a06cee"
server_key_iv: "cafa39e23453980caf05f23afbc4259b"
server_key_version: "encryptionkey@0"
I20260927 05:11:06.782341 20267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:06.831916 20267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:06.832964 20267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:06.833285 20267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:06.853072 20267 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44255
I20260927 05:11:06.853103 20332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44255 every 8 connection(s)
I20260927 05:11:06.854707 20267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:06.858881 20333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:06.859524 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 20267
I20260927 05:11:06.859696 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:06.860333 388 external_mini_cluster.cc:1428] Setting key ad0272eac00b391510aeff829b8a46c4
I20260927 05:11:06.873639 20333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328: Bootstrap starting.
I20260927 05:11:06.876258 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:06.862496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.0.97.62:44255)
0927 05:11:06.863673 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:06.863686 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:06.863741 (+ 55us) server_negotiation.cc:407] Connection header received
0927 05:11:06.866253 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:06.866285 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:06.866913 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:06.867391 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:06.868084 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:06.869600 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:06.871114 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:06.871913 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:06.872190 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:06.873575 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:06.873584 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:06.873601 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:06.874277 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:06.874384 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:06.874409 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:06.874552 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:06.874801 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 05:11:06.875153 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":413,"threads_started":1}
I20260927 05:11:06.877275 20333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:06.878703 20333 log.cc:824] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:06.881938 20333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328: No bootstrap required, opened a new log
I20260927 05:11:06.890358 20333 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56fda272b747e9a59ee1c83eb77328" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44255 } }
I20260927 05:11:06.890833 20333 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:06.890920 20333 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef56fda272b747e9a59ee1c83eb77328, State: Initialized, Role: FOLLOWER
I20260927 05:11:06.891541 20333 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56fda272b747e9a59ee1c83eb77328" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44255 } }
I20260927 05:11:06.891827 20333 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:06.891999 20333 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:06.892215 20333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:06.894781 20333 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56fda272b747e9a59ee1c83eb77328" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44255 } }
I20260927 05:11:06.895440 20333 leader_election.cc:302] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef56fda272b747e9a59ee1c83eb77328; no voters:
I20260927 05:11:06.896179 20333 leader_election.cc:288] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:06.896339 20338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:06.897289 20338 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [term 1 LEADER]: Becoming Leader. State: Replica: ef56fda272b747e9a59ee1c83eb77328, State: Running, Role: LEADER
I20260927 05:11:06.897946 20338 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56fda272b747e9a59ee1c83eb77328" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44255 } }
I20260927 05:11:06.898474 20333 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:06.900825 20340 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef56fda272b747e9a59ee1c83eb77328. Latest consensus state: current_term: 1 leader_uuid: "ef56fda272b747e9a59ee1c83eb77328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56fda272b747e9a59ee1c83eb77328" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44255 } } }
I20260927 05:11:06.901316 20340 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:06.901154 20339 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef56fda272b747e9a59ee1c83eb77328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56fda272b747e9a59ee1c83eb77328" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44255 } } }
I20260927 05:11:06.901572 20339 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:06.904505 20344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:06.911502 20344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:06.922642 20344 catalog_manager.cc:1347] Generated new cluster ID: 5359644e88e44902a5e455c14109d1f1
I20260927 05:11:06.922793 20344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:06.930670 20344 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:06.932720 20344 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:06.940984 20344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328: Generated new TSK 0
I20260927 05:11:06.941998 20344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:06.949122 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44255
--builtin_ntp_servers=127.0.97.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 05:11:07.329317 20357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:07.329643 20357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:07.329818 20357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:07.337659 20357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:07.337781 20357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:07.338024 20357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260927 05:11:07.348137 20357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:07.349522 20357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:07.351483 20357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:07.366331 20362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:07.366367 20365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:07.366415 20363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:07.367350 20357 server_base.cc:1034] running on GCE node
I20260927 05:11:07.368149 20357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:07.369258 20357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:07.370496 20357 hybrid_clock.cc:648] HybridClock initialized: now 1790485867370431 us; error 43 us; skew 500 ppm
I20260927 05:11:07.370908 20357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:11:07.373572 20357 webserver.cc:460] Webserver started at http://127.0.97.1:39759/ using document root <none> and password file <none>
I20260927 05:11:07.374400 20357 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:07.374537 20357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:07.374909 20357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:07.378427 20357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/data/instance:
uuid: "a9e581d8c4d541409e093faacdfafda7"
format_stamp: "Formatted at 2026-09-27 05:11:07 on dist-test-slave-hgs0"
server_key: "b3dedc1029a64a4bab20423f7282073f"
server_key_iv: "abab731b3df0b2f424cab17b5f1dabf3"
server_key_version: "encryptionkey@0"
I20260927 05:11:07.379347 20357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a9e581d8c4d541409e093faacdfafda7"
format_stamp: "Formatted at 2026-09-27 05:11:07 on dist-test-slave-hgs0"
server_key: "b3dedc1029a64a4bab20423f7282073f"
server_key_iv: "abab731b3df0b2f424cab17b5f1dabf3"
server_key_version: "encryptionkey@0"
I20260927 05:11:07.385720 20357 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 05:11:07.389984 20371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:07.391641 20357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:11:07.391860 20357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/wal
uuid: "a9e581d8c4d541409e093faacdfafda7"
format_stamp: "Formatted at 2026-09-27 05:11:07 on dist-test-slave-hgs0"
server_key: "b3dedc1029a64a4bab20423f7282073f"
server_key_iv: "abab731b3df0b2f424cab17b5f1dabf3"
server_key_version: "encryptionkey@0"
I20260927 05:11:07.392186 20357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:07.416376 20357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:07.417337 20357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:07.417654 20357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:07.418740 20357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:11:07.420405 20357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:11:07.420529 20357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:07.420689 20357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:11:07.420784 20357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:07.454147 20357 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36171
I20260927 05:11:07.454242 20483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36171 every 8 connection(s)
I20260927 05:11:07.455588 20357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/data/info.pb
I20260927 05:11:07.458158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 20357
I20260927 05:11:07.458320 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-0/wal/instance
I20260927 05:11:07.458868 388 external_mini_cluster.cc:1428] Setting key 99f4f63a038c6061810a681558a82d15
I20260927 05:11:07.461836 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44255
--builtin_ntp_servers=127.0.97.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:11:07.470383 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:07.458667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56853 (local address 127.0.97.62:44255)
0927 05:11:07.459220 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:07.459227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:07.460975 (+ 1748us) server_negotiation.cc:407] Connection header received
0927 05:11:07.463183 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:07.463189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:07.463292 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:07.463435 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:07.464737 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:07.465553 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:07.467287 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:07.467733 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:07.467887 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:07.469452 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:07.469459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:07.469463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:07.469484 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:07.469526 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:07.469538 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:07.469641 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:07.469880 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 05:11:07.470000 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
I20260927 05:11:07.471596 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:07.459421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44255 (local address 127.0.97.1:56853)
0927 05:11:07.460785 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:07.460843 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:07.462864 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:07.463716 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:07.463726 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:07.463833 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:07.464550 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:07.464568 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:07.465747 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:07.465751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:07.467131 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:07.467143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:07.468022 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:07.468032 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:07.468292 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:07.469083 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:11:07.469085 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:07.469130 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:11:07.469219 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:07.469641 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:07.469655 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:07.469665 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:11:07.469853 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 05:11:07.470311 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":478,"threads_started":1}
I20260927 05:11:07.473260 20484 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44255
I20260927 05:11:07.473618 20484 heartbeater.cc:464] Registering TS with master...
I20260927 05:11:07.474570 20484 heartbeater.cc:511] Master 127.0.97.62:44255 requested a full tablet report, sending...
I20260927 05:11:07.476992 20298 ts_manager.cc:194] Registered new tserver with Master: a9e581d8c4d541409e093faacdfafda7 (127.0.97.1:36171)
I20260927 05:11:07.478538 20298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:56853
W20260927 05:11:07.811189 20488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:07.811498 20488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:07.811652 20488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:07.819905 20488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:07.820113 20488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:07.820307 20488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260927 05:11:07.830014 20488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:07.831476 20488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:07.833647 20488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:07.848351 20493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:07.848400 20496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:07.848366 20494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:07.849380 20488 server_base.cc:1034] running on GCE node
I20260927 05:11:07.850060 20488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:07.851186 20488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:07.852411 20488 hybrid_clock.cc:648] HybridClock initialized: now 1790485867852355 us; error 60 us; skew 500 ppm
I20260927 05:11:07.852823 20488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:11:07.855599 20488 webserver.cc:460] Webserver started at http://127.0.97.2:39561/ using document root <none> and password file <none>
I20260927 05:11:07.856494 20488 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:07.856629 20488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:07.857102 20488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:07.860826 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/data/instance:
uuid: "9a98c9ee53a6485097857bc8add7f919"
format_stamp: "Formatted at 2026-09-27 05:11:07 on dist-test-slave-hgs0"
server_key: "39ca7b98551b8ca95bdf0852b9f34c14"
server_key_iv: "873c82d3f2e1dc7f2042d568c21cbda5"
server_key_version: "encryptionkey@0"
I20260927 05:11:07.861786 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9a98c9ee53a6485097857bc8add7f919"
format_stamp: "Formatted at 2026-09-27 05:11:07 on dist-test-slave-hgs0"
server_key: "39ca7b98551b8ca95bdf0852b9f34c14"
server_key_iv: "873c82d3f2e1dc7f2042d568c21cbda5"
server_key_version: "encryptionkey@0"
I20260927 05:11:07.867694 20488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:11:07.871767 20502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:07.873458 20488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:11:07.873667 20488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/wal
uuid: "9a98c9ee53a6485097857bc8add7f919"
format_stamp: "Formatted at 2026-09-27 05:11:07 on dist-test-slave-hgs0"
server_key: "39ca7b98551b8ca95bdf0852b9f34c14"
server_key_iv: "873c82d3f2e1dc7f2042d568c21cbda5"
server_key_version: "encryptionkey@0"
I20260927 05:11:07.873950 20488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:07.886325 20488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:07.887364 20488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:07.887665 20488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:07.888859 20488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:11:07.890482 20488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:11:07.890604 20488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:07.890748 20488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:11:07.890833 20488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:07.924763 20488 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36021
I20260927 05:11:07.924811 20614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36021 every 8 connection(s)
I20260927 05:11:07.926230 20488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/data/info.pb
I20260927 05:11:07.930388 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 20488
I20260927 05:11:07.930574 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/wal/instance
I20260927 05:11:07.931161 388 external_mini_cluster.cc:1428] Setting key 13e051b27f31a68371f5227893d9663e
I20260927 05:11:07.934358 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44255
--builtin_ntp_servers=127.0.97.20:40051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 05:11:07.941152 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:07.929225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55159 (local address 127.0.97.62:44255)
0927 05:11:07.929445 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:07.929449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:07.931145 (+ 1696us) server_negotiation.cc:407] Connection header received
0927 05:11:07.933410 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:07.933417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:07.933503 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:07.933630 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:07.934871 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:07.936137 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:07.937759 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:07.938490 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:07.938646 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:07.940164 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:07.940174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:07.940182 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:07.940210 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:07.940255 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:07.940273 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:07.940367 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:07.940606 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 05:11:07.940727 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 05:11:07.942258 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:07.929784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44255 (local address 127.0.97.2:55159)
0927 05:11:07.930968 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:07.931009 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:07.933054 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:07.933846 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:07.933854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:07.933965 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:07.934686 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:07.934708 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:07.936284 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:07.936289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:07.937598 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:07.937610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:07.938741 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:07.938746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:07.938982 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:07.939756 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:11:07.939759 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:07.939806 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:11:07.939893 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:07.940376 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:07.940385 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:07.940391 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 05:11:07.940566 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 05:11:07.940895 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":378,"threads_started":1}
I20260927 05:11:07.943980 20615 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44255
I20260927 05:11:07.944321 20615 heartbeater.cc:464] Registering TS with master...
I20260927 05:11:07.945128 20615 heartbeater.cc:511] Master 127.0.97.62:44255 requested a full tablet report, sending...
I20260927 05:11:07.946660 20298 ts_manager.cc:194] Registered new tserver with Master: 9a98c9ee53a6485097857bc8add7f919 (127.0.97.2:36021)
I20260927 05:11:07.947491 20298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:55159
W20260927 05:11:08.303507 20618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:08.303840 20618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:08.304085 20618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:08.312074 20618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:08.312201 20618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:08.312412 20618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260927 05:11:08.322365 20618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:08.323808 20618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:08.325845 20618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:08.341262 20623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:08.341305 20624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:08.341329 20626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:08.342900 20618 server_base.cc:1034] running on GCE node
I20260927 05:11:08.343803 20618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:08.345227 20618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:08.346498 20618 hybrid_clock.cc:648] HybridClock initialized: now 1790485868346416 us; error 76 us; skew 500 ppm
I20260927 05:11:08.346894 20618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:11:08.349475 20618 webserver.cc:460] Webserver started at http://127.0.97.3:37743/ using document root <none> and password file <none>
I20260927 05:11:08.350270 20618 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:08.350426 20618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:08.350788 20618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:08.354038 20618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/data/instance:
uuid: "abda6bd19d6c46dda50575189c1bf6a9"
format_stamp: "Formatted at 2026-09-27 05:11:08 on dist-test-slave-hgs0"
server_key: "1717592e3cf4158a8d81d4b167459eae"
server_key_iv: "6c910825470a7eab163fa0974d8d9157"
server_key_version: "encryptionkey@0"
I20260927 05:11:08.354952 20618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/wal/instance:
uuid: "abda6bd19d6c46dda50575189c1bf6a9"
format_stamp: "Formatted at 2026-09-27 05:11:08 on dist-test-slave-hgs0"
server_key: "1717592e3cf4158a8d81d4b167459eae"
server_key_iv: "6c910825470a7eab163fa0974d8d9157"
server_key_version: "encryptionkey@0"
I20260927 05:11:08.360627 20618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 05:11:08.364648 20632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:08.366139 20618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:11:08.366338 20618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/wal
uuid: "abda6bd19d6c46dda50575189c1bf6a9"
format_stamp: "Formatted at 2026-09-27 05:11:08 on dist-test-slave-hgs0"
server_key: "1717592e3cf4158a8d81d4b167459eae"
server_key_iv: "6c910825470a7eab163fa0974d8d9157"
server_key_version: "encryptionkey@0"
I20260927 05:11:08.366601 20618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:08.384681 20618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:08.385921 20618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:08.386251 20618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:08.387599 20618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 05:11:08.389636 20618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 05:11:08.389756 20618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:08.389878 20618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 05:11:08.389953 20618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:08.427467 20618 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43667
I20260927 05:11:08.427507 20744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43667 every 8 connection(s)
I20260927 05:11:08.428990 20618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/data/info.pb
I20260927 05:11:08.434244 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 20618
I20260927 05:11:08.434450 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/wal/instance
I20260927 05:11:08.435024 388 external_mini_cluster.cc:1428] Setting key 3d3d730416de3fa0a7abfe9b4d6fb484
I20260927 05:11:08.443539 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.431690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39187 (local address 127.0.97.62:44255)
0927 05:11:08.431864 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:08.431868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:08.433495 (+ 1627us) server_negotiation.cc:407] Connection header received
0927 05:11:08.436149 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:08.436155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:08.436263 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:08.436418 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:08.437836 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.438917 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.440633 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.441079 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.441217 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.442663 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:08.442669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:08.442672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:08.442692 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:08.442725 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:08.442738 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:08.442837 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:08.443076 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 05:11:08.443195 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:11:08.444756 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.432350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44255 (local address 127.0.97.3:39187)
0927 05:11:08.433325 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:08.433363 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:08.435768 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:08.436730 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:08.436744 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:08.436888 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:08.437656 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.437675 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.439075 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.439079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.440472 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.440486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.441296 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.441300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.441520 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.442300 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 05:11:08.442303 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:08.442349 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 05:11:08.442439 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:08.442838 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:08.442851 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:08.442861 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 05:11:08.443059 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 05:11:08.443529 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":302,"threads_started":1}
I20260927 05:11:08.446424 20745 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44255
I20260927 05:11:08.446756 20745 heartbeater.cc:464] Registering TS with master...
I20260927 05:11:08.447566 20745 heartbeater.cc:511] Master 127.0.97.62:44255 requested a full tablet report, sending...
I20260927 05:11:08.449275 20298 ts_manager.cc:194] Registered new tserver with Master: abda6bd19d6c46dda50575189c1bf6a9 (127.0.97.3:43667)
I20260927 05:11:08.450438 20298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:39187
I20260927 05:11:08.451221 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 05:11:08.462848 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.455911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.0.97.62:44255)
0927 05:11:08.456185 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:08.456192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:08.456670 (+ 478us) server_negotiation.cc:407] Connection header received
0927 05:11:08.456797 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:08.456804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:08.456892 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:08.457011 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:08.457511 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.458745 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.460403 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.461003 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.461114 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.462229 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:08.462233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:08.462236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:08.462255 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:08.462285 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:08.462296 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:08.462366 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:08.462516 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 05:11:08.462601 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 05:11:08.467022 20298 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 05:11:08.474149 20298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 05:11:08.482703 20484 heartbeater.cc:503] Master 127.0.97.62:44255 was elected leader, sending a full tablet report...
I20260927 05:11:08.500483 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.488475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36021 (local address 127.0.0.1:50518)
0927 05:11:08.489652 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:08.489693 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:08.489873 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:08.491100 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:08.491111 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:08.491154 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:08.491595 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.491619 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.494146 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.494152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.497009 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.497025 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.499084 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.499092 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.499420 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.499494 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 05:11:08.499650 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 05:11:08.499982 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":5,"thread_start_us":786,"threads_started":1}
I20260927 05:11:08.501142 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.490212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36171 (local address 127.0.0.1:40738)
0927 05:11:08.490835 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:08.490859 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:08.490999 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:08.492318 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:08.492323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:08.492352 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:08.492878 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.492895 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.495284 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.495296 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.497890 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.497904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.500187 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.500195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.500429 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.500481 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 05:11:08.500581 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 05:11:08.500734 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":33,"thread_start_us":301,"threads_started":1}
I20260927 05:11:08.501744 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.488445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.0.97.2:36021)
0927 05:11:08.489130 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:08.489138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:08.489836 (+ 698us) server_negotiation.cc:407] Connection header received
0927 05:11:08.490125 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:08.490135 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:08.490481 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:08.490742 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:08.491845 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.493930 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.497279 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.498866 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.499016 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.499298 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:08.501127 (+ 1829us) server_negotiation.cc:299] Negotiation successful
0927 05:11:08.501359 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":349,"threads_started":1}
I20260927 05:11:08.502138 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.490613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.0.97.1:36171)
0927 05:11:08.491583 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:08.491593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:08.491627 (+ 34us) server_negotiation.cc:407] Connection header received
0927 05:11:08.491740 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:08.491748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:08.492075 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:08.493163 (+ 1088us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:08.493493 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.495058 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.498112 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.499544 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.499686 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.500025 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:08.501536 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0927 05:11:08.501762 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":1804,"threads_started":1}
I20260927 05:11:08.503737 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.491797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43667 (local address 127.0.0.1:57802)
0927 05:11:08.493751 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:08.493773 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:08.493938 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:08.494678 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:08.494684 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:08.494714 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:08.495230 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.495247 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.497616 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.497622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.499843 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.499859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.502100 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.502106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.503117 (+ 1011us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.503152 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 05:11:08.503245 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 05:11:08.503382 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1812,"mutex_wait_us":54,"thread_start_us":1647,"threads_started":1}
I20260927 05:11:08.503885 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.491298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.0.97.3:43667)
0927 05:11:08.493261 (+ 1963us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:08.493271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:08.493914 (+ 643us) server_negotiation.cc:407] Connection header received
0927 05:11:08.494114 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:08.494126 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:08.494432 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:08.494663 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:08.495496 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.497449 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.500155 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.501739 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.502976 (+ 1237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.503271 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:08.503336 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 05:11:08.503494 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1766,"thread_start_us":1558,"threads_started":1}
I20260927 05:11:08.506322 20419 tablet_service.cc:1467] Processing CreateTablet for tablet 89bd61966e1343a3af5f470547d4729e (DEFAULT_TABLE table=test-table [id=909b8e3323144e99ac85675788ed1970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:11:08.506322 20550 tablet_service.cc:1467] Processing CreateTablet for tablet 89bd61966e1343a3af5f470547d4729e (DEFAULT_TABLE table=test-table [id=909b8e3323144e99ac85675788ed1970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:11:08.506850 20680 tablet_service.cc:1467] Processing CreateTablet for tablet 89bd61966e1343a3af5f470547d4729e (DEFAULT_TABLE table=test-table [id=909b8e3323144e99ac85675788ed1970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 05:11:08.508014 20680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89bd61966e1343a3af5f470547d4729e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:08.508034 20550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89bd61966e1343a3af5f470547d4729e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:08.508041 20419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89bd61966e1343a3af5f470547d4729e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:08.519989 20763 tablet_bootstrap.cc:492] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: Bootstrap starting.
I20260927 05:11:08.524067 20763 tablet_bootstrap.cc:654] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:08.525257 20763 log.cc:824] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:08.526944 20764 tablet_bootstrap.cc:492] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7: Bootstrap starting.
I20260927 05:11:08.529132 20763 tablet_bootstrap.cc:492] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: No bootstrap required, opened a new log
I20260927 05:11:08.529454 20763 ts_tablet_manager.cc:1397] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260927 05:11:08.531164 20764 tablet_bootstrap.cc:654] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:08.532555 20764 log.cc:824] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:08.533921 20765 tablet_bootstrap.cc:492] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: Bootstrap starting.
I20260927 05:11:08.537148 20764 tablet_bootstrap.cc:492] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7: No bootstrap required, opened a new log
I20260927 05:11:08.537460 20764 ts_tablet_manager.cc:1397] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20260927 05:11:08.537806 20765 tablet_bootstrap.cc:654] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:08.539139 20765 log.cc:824] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:08.540673 20763 raft_consensus.cc:348] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a98c9ee53a6485097857bc8add7f919" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36021 } } peers { permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 } } peers { permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 } }
I20260927 05:11:08.541126 20763 raft_consensus.cc:374] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:08.541255 20763 raft_consensus.cc:729] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abda6bd19d6c46dda50575189c1bf6a9, State: Initialized, Role: FOLLOWER
I20260927 05:11:08.542115 20763 consensus_queue.cc:260] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a98c9ee53a6485097857bc8add7f919" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36021 } } peers { permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 } } peers { permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 } }
I20260927 05:11:08.545073 20745 heartbeater.cc:503] Master 127.0.97.62:44255 was elected leader, sending a full tablet report...
I20260927 05:11:08.545222 20765 tablet_bootstrap.cc:492] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: No bootstrap required, opened a new log
I20260927 05:11:08.545559 20765 ts_tablet_manager.cc:1397] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260927 05:11:08.545971 20763 ts_tablet_manager.cc:1428] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260927 05:11:08.548586 20764 raft_consensus.cc:348] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a98c9ee53a6485097857bc8add7f919" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36021 } } peers { permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 } } peers { permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 } }
I20260927 05:11:08.549028 20764 raft_consensus.cc:374] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:08.549178 20764 raft_consensus.cc:729] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e581d8c4d541409e093faacdfafda7, State: Initialized, Role: FOLLOWER
I20260927 05:11:08.549983 20764 consensus_queue.cc:260] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a98c9ee53a6485097857bc8add7f919" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36021 } } peers { permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 } } peers { permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 } }
I20260927 05:11:08.552695 20764 ts_tablet_manager.cc:1428] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260927 05:11:08.556430 20765 raft_consensus.cc:348] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a98c9ee53a6485097857bc8add7f919" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36021 } } peers { permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 } } peers { permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 } }
I20260927 05:11:08.557036 20765 raft_consensus.cc:374] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:08.557158 20765 raft_consensus.cc:729] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a98c9ee53a6485097857bc8add7f919, State: Initialized, Role: FOLLOWER
I20260927 05:11:08.557814 20765 consensus_queue.cc:260] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a98c9ee53a6485097857bc8add7f919" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36021 } } peers { permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 } } peers { permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 } }
I20260927 05:11:08.558947 20615 heartbeater.cc:503] Master 127.0.97.62:44255 was elected leader, sending a full tablet report...
I20260927 05:11:08.559549 20765 ts_tablet_manager.cc:1428] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
W20260927 05:11:08.678498 20616 tablet.cc:2369] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:11:08.680701 20746 tablet.cc:2369] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:11:08.708498 20485 tablet.cc:2369] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:11:08.744794 20771 raft_consensus.cc:482] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 05:11:08.745095 20771 raft_consensus.cc:504] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a98c9ee53a6485097857bc8add7f919" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36021 } } peers { permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 } } peers { permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 } }
I20260927 05:11:08.746811 20771 leader_election.cc:288] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9e581d8c4d541409e093faacdfafda7 (127.0.97.1:36171), abda6bd19d6c46dda50575189c1bf6a9 (127.0.97.3:43667)
I20260927 05:11:08.754791 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.747256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40047 (local address 127.0.97.1:36171)
0927 05:11:08.747545 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:08.747555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:08.748067 (+ 512us) server_negotiation.cc:407] Connection header received
0927 05:11:08.748326 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:08.748334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:08.748454 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:08.748566 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:08.749195 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.750184 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.752778 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.753756 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.753891 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.754019 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:08.754396 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 05:11:08.754538 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:11:08.754839 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.747190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36171 (local address 127.0.97.2:40047)
0927 05:11:08.747882 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:08.747902 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:08.748136 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:08.748589 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:08.748594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:08.748622 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:08.749052 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.749072 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.750319 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.750323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.752581 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.752599 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.753972 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.753982 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.754199 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.754236 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 05:11:08.754312 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 05:11:08.754460 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":406,"threads_started":1}
I20260927 05:11:08.754840 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.748216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46641 (local address 127.0.97.3:43667)
0927 05:11:08.748448 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:08.748456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:08.748862 (+ 406us) server_negotiation.cc:407] Connection header received
0927 05:11:08.749040 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:08.749044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:08.749104 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:08.749217 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 05:11:08.749689 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.750577 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.752999 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.753915 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.754045 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.754147 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:08.754486 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 05:11:08.754622 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:11:08.754875 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.748091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43667 (local address 127.0.97.2:46641)
0927 05:11:08.748744 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:08.748764 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:08.748921 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:08.749244 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:08.749247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:08.749268 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 05:11:08.749548 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.749555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.750704 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.750709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.752809 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:08.752819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.754128 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.754134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:08.754264 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.754310 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 05:11:08.754414 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 05:11:08.754538 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":331,"threads_started":1}
I20260927 05:11:08.755993 20700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bd61966e1343a3af5f470547d4729e" candidate_uuid: "9a98c9ee53a6485097857bc8add7f919" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abda6bd19d6c46dda50575189c1bf6a9" is_pre_election: true
I20260927 05:11:08.756006 20439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bd61966e1343a3af5f470547d4729e" candidate_uuid: "9a98c9ee53a6485097857bc8add7f919" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e581d8c4d541409e093faacdfafda7" is_pre_election: true
I20260927 05:11:08.756539 20700 raft_consensus.cc:2454] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a98c9ee53a6485097857bc8add7f919 in term 0.
I20260927 05:11:08.756543 20439 raft_consensus.cc:2454] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a98c9ee53a6485097857bc8add7f919 in term 0.
I20260927 05:11:08.757202 20504 leader_election.cc:302] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a98c9ee53a6485097857bc8add7f919, abda6bd19d6c46dda50575189c1bf6a9; no voters:
I20260927 05:11:08.757618 20771 raft_consensus.cc:2781] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 05:11:08.757746 20771 raft_consensus.cc:482] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 05:11:08.757893 20771 raft_consensus.cc:3037] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:08.760203 20771 raft_consensus.cc:504] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a98c9ee53a6485097857bc8add7f919" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36021 } } peers { permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 } } peers { permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 } }
I20260927 05:11:08.760850 20771 leader_election.cc:288] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [CANDIDATE]: Term 1 election: Requested vote from peers a9e581d8c4d541409e093faacdfafda7 (127.0.97.1:36171), abda6bd19d6c46dda50575189c1bf6a9 (127.0.97.3:43667)
I20260927 05:11:08.761550 20439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bd61966e1343a3af5f470547d4729e" candidate_uuid: "9a98c9ee53a6485097857bc8add7f919" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e581d8c4d541409e093faacdfafda7"
I20260927 05:11:08.761606 20700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bd61966e1343a3af5f470547d4729e" candidate_uuid: "9a98c9ee53a6485097857bc8add7f919" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abda6bd19d6c46dda50575189c1bf6a9"
I20260927 05:11:08.761822 20700 raft_consensus.cc:3037] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:08.761817 20439 raft_consensus.cc:3037] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:08.764343 20700 raft_consensus.cc:2454] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a98c9ee53a6485097857bc8add7f919 in term 1.
I20260927 05:11:08.764463 20439 raft_consensus.cc:2454] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a98c9ee53a6485097857bc8add7f919 in term 1.
I20260927 05:11:08.764925 20504 leader_election.cc:302] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a98c9ee53a6485097857bc8add7f919, abda6bd19d6c46dda50575189c1bf6a9; no voters:
I20260927 05:11:08.765245 20771 raft_consensus.cc:2781] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:08.765780 20771 raft_consensus.cc:686] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 1 LEADER]: Becoming Leader. State: Replica: 9a98c9ee53a6485097857bc8add7f919, State: Running, Role: LEADER
I20260927 05:11:08.766425 20771 consensus_queue.cc:237] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a98c9ee53a6485097857bc8add7f919" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36021 } } peers { permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 } } peers { permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 } }
I20260927 05:11:08.771785 20298 catalog_manager.cc:5495] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a98c9ee53a6485097857bc8add7f919 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9a98c9ee53a6485097857bc8add7f919" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a98c9ee53a6485097857bc8add7f919" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 } health_report { overall_health: UNKNOWN } } }
I20260927 05:11:08.801890 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.796214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.0.97.2:36021)
0927 05:11:08.796435 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:08.796440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:08.796460 (+ 20us) server_negotiation.cc:407] Connection header received
0927 05:11:08.796609 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:08.796613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:08.796710 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:08.796911 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:08.797340 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.798132 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.799730 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.800222 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.800328 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.800826 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:08.801344 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:08.801455 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:08.801591 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 05:11:08.801674 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:11:08.811458 20700 raft_consensus.cc:1261] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9 [term 1 FOLLOWER]: Refusing update from remote peer 9a98c9ee53a6485097857bc8add7f919: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:11:08.811532 20439 raft_consensus.cc:1261] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7 [term 1 FOLLOWER]: Refusing update from remote peer 9a98c9ee53a6485097857bc8add7f919: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:11:08.812724 20774 consensus_queue.cc:1035] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e581d8c4d541409e093faacdfafda7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:11:08.813131 20771 consensus_queue.cc:1035] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abda6bd19d6c46dda50575189c1bf6a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:11:08.842183 20297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:11:08.842533 20297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 05:11:08.846751 20297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef56fda272b747e9a59ee1c83eb77328: Sending DeleteTablet for 3 replicas of tablet 89bd61966e1343a3af5f470547d4729e
I20260927 05:11:08.848196 20550 tablet_service.cc:1514] Processing DeleteTablet for tablet 89bd61966e1343a3af5f470547d4729e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:11:08 UTC) from {username='slave'} at 127.0.0.1:50518
I20260927 05:11:08.848603 20680 tablet_service.cc:1514] Processing DeleteTablet for tablet 89bd61966e1343a3af5f470547d4729e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:11:08 UTC) from {username='slave'} at 127.0.0.1:57802
I20260927 05:11:08.848649 20419 tablet_service.cc:1514] Processing DeleteTablet for tablet 89bd61966e1343a3af5f470547d4729e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 05:11:08 UTC) from {username='slave'} at 127.0.0.1:40738
I20260927 05:11:08.850503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 20357
I20260927 05:11:08.868477 20788 tablet_replica.cc:330] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: stopping tablet replica
I20260927 05:11:08.869030 20788 raft_consensus.cc:2229] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:11:08.869589 20788 raft_consensus.cc:2258] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:11:08.872191 20788 ts_tablet_manager.cc:1905] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:11:08.878422 20787 tablet_replica.cc:330] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: stopping tablet replica
I20260927 05:11:08.879071 20787 raft_consensus.cc:2229] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 1 LEADER]: Raft consensus shutting down.
I20260927 05:11:08.878325 20789 tablet_replica.cc:330] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7: stopping tablet replica
I20260927 05:11:08.879703 20789 raft_consensus.cc:2229] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 05:11:08.879806 20788 ts_tablet_manager.cc:1918] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:11:08.880004 20788 log.cc:1196] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-2/wal/wals/89bd61966e1343a3af5f470547d4729e
I20260927 05:11:08.880582 20788 ts_tablet_manager.cc:1939] T 89bd61966e1343a3af5f470547d4729e P abda6bd19d6c46dda50575189c1bf6a9: Deleting consensus metadata
I20260927 05:11:08.881188 20789 raft_consensus.cc:2258] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:11:08.881608 20787 raft_consensus.cc:2258] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 05:11:08.882681 20283 catalog_manager.cc:4841] TS abda6bd19d6c46dda50575189c1bf6a9 (127.0.97.3:43667): tablet 89bd61966e1343a3af5f470547d4729e (table test-table [id=909b8e3323144e99ac85675788ed1970]) successfully deleted
I20260927 05:11:08.884191 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:08.852506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.0.97.1:36171)
0927 05:11:08.852796 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:08.852807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:08.871543 (+ 18736us) server_negotiation.cc:407] Connection header received
0927 05:11:08.871808 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:08.871819 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:08.872062 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:08.872231 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:08.873341 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.875425 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.878724 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:08.879555 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:08.879674 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:08.881736 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:08.881748 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:08.881767 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:08.882614 (+ 847us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:08.882682 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:08.882714 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:08.882817 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:08.883619 (+ 802us) server_negotiation.cc:299] Negotiation successful
0927 05:11:08.883764 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 05:11:08.885495 20479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:08.886441 20787 ts_tablet_manager.cc:1905] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:11:08.897411 20787 ts_tablet_manager.cc:1918] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 05:11:08.897670 20787 log.cc:1196] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: Deleting WAL directory at /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790485535376807-388-0/minicluster-data/ts-1/wal/wals/89bd61966e1343a3af5f470547d4729e
I20260927 05:11:08.898320 20787 ts_tablet_manager.cc:1939] T 89bd61966e1343a3af5f470547d4729e P 9a98c9ee53a6485097857bc8add7f919: Deleting consensus metadata
I20260927 05:11:08.900689 20283 catalog_manager.cc:4841] TS 9a98c9ee53a6485097857bc8add7f919 (127.0.97.2:36021): tablet 89bd61966e1343a3af5f470547d4729e (table test-table [id=909b8e3323144e99ac85675788ed1970]) successfully deleted
I20260927 05:11:08.886224 20789 ts_tablet_manager.cc:1905] T 89bd61966e1343a3af5f470547d4729e P a9e581d8c4d541409e093faacdfafda7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 05:11:08.982460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 20357
W20260927 05:11:09.001097 20283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36171 (error 108)
I20260927 05:11:09.001581 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 20488
I20260927 05:11:09.003130 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:09.002493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36171 (local address 127.0.0.1:40758)
0927 05:11:09.002762 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:09.002932 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260927 05:11:09.003415 20283 catalog_manager.cc:4584] TS a9e581d8c4d541409e093faacdfafda7 (127.0.97.1:36171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89bd61966e1343a3af5f470547d4729e: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36171: connect: Connection refused (error 111)
I20260927 05:11:09.012097 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:09.002815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.0.97.2:36021)
0927 05:11:09.003016 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:09.003022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:09.003041 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:11:09.003121 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:09.003132 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:09.003247 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:09.003365 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:09.003897 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:09.004831 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:09.006900 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:09.007778 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:09.007962 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:09.009933 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:09.009950 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:09.009973 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:09.010986 (+ 1013us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:09.011065 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:09.011095 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:09.011218 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:09.011436 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 05:11:09.011625 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:11:09.013296 20610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:09.057159 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:09.056642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36171 (local address 127.0.0.1:40762)
0927 05:11:09.056883 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:09.056993 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 05:11:09.058416 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:09.058003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36171 (local address 127.0.0.1:40766)
0927 05:11:09.058181 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:09.058271 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 05:11:09.119478 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:09.118825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36171 (local address 127.0.0.1:40776)
0927 05:11:09.119105 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:09.119208 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 05:11:09.120736 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:09.120341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36171 (local address 127.0.0.1:40784)
0927 05:11:09.120503 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:09.120600 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 05:11:09.128756 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 20488
I20260927 05:11:09.149617 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 20618
I20260927 05:11:09.157070 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:09.150616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.0.97.3:43667)
0927 05:11:09.150803 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:09.150808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:09.150831 (+ 23us) server_negotiation.cc:407] Connection header received
0927 05:11:09.150948 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:09.150952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:09.151038 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:09.151145 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:09.151512 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:09.152467 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:09.153942 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:09.154344 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:09.154454 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:09.155552 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:09.155563 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:09.155579 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:09.156337 (+ 758us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:09.156394 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:09.156416 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:09.156506 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:09.156643 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 05:11:09.156751 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 05:11:09.157821 20740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:09.201774 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:09.201274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36171 (local address 127.0.0.1:40798)
0927 05:11:09.201485 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:09.201583 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 05:11:09.202961 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:09.202505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36171 (local address 127.0.0.1:40802)
0927 05:11:09.202739 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:09.202821 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 05:11:09.255604 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 20618
I20260927 05:11:09.273308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 20267
I20260927 05:11:09.274600 20328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:09.387995 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:09.387233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36171 (local address 127.0.0.1:40810)
0927 05:11:09.387514 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:09.387659 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 05:11:09.388242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 20267
2026-09-27T05:11:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3117 ms)
[----------] 6 tests from EncryptionPolicyTest (23101 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:09 dist-test-slave-hgs0 krb5kdc[20798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:09 dist-test-slave-hgs0 krb5kdc[20798](info): set up 2 sockets
Sep 27 05:11:09 dist-test-slave-hgs0 krb5kdc[20798](info): commencing operation
krb5kdc: starting...
W20260927 05:11:10.552549 388 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.005s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:10 dist-test-slave-hgs0 krb5kdc[20798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:10.840538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:11.204778 20811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:11.205116 20811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:11.205296 20811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:11.213229 20811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:11.213405 20811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:11.213500 20811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:11.213624 20811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:11.213713 20811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:11.224838 20811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:11.226315 20811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:11.227811 20811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:11.240943 20819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:11.240986 20817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:11.241115 20816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:11.242242 20811 server_base.cc:1034] running on GCE node
I20260927 05:11:11.243170 20811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:11.245013 20811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:11.246244 20811 hybrid_clock.cc:648] HybridClock initialized: now 1790485871246165 us; error 72 us; skew 500 ppm
I20260927 05:11:11.246624 20811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:11 dist-test-slave-hgs0 krb5kdc[20798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:11.250469 20811 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:11.252235 20811 webserver.cc:460] Webserver started at http://127.0.0.1:37895/ using document root <none> and password file <none>
I20260927 05:11:11.253028 20811 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:11.253175 20811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:11.253629 20811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:11.256959 20811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "419f5bc1205a40e8aab2636d1841537d"
format_stamp: "Formatted at 2026-09-27 05:11:11 on dist-test-slave-hgs0"
server_key: "a217a90481d2b94fbca74435fdd508ad"
server_key_iv: "00ad978da6083a86b0abec3411d74faa"
server_key_version: "encryptionkey@0"
I20260927 05:11:11.257797 20811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "419f5bc1205a40e8aab2636d1841537d"
format_stamp: "Formatted at 2026-09-27 05:11:11 on dist-test-slave-hgs0"
server_key: "a217a90481d2b94fbca74435fdd508ad"
server_key_iv: "00ad978da6083a86b0abec3411d74faa"
server_key_version: "encryptionkey@0"
I20260927 05:11:11.263513 20811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260927 05:11:11.267539 20826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:11.269173 20811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:11:11.269399 20811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "419f5bc1205a40e8aab2636d1841537d"
format_stamp: "Formatted at 2026-09-27 05:11:11 on dist-test-slave-hgs0"
server_key: "a217a90481d2b94fbca74435fdd508ad"
server_key_iv: "00ad978da6083a86b0abec3411d74faa"
server_key_version: "encryptionkey@0"
I20260927 05:11:11.269611 20811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:11.296665 20811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:11.301836 20811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:11.302174 20811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:11.323064 20811 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33515
I20260927 05:11:11.323114 20877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33515 every 8 connection(s)
I20260927 05:11:11.324864 20811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:11.328243 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 20811
I20260927 05:11:11.328471 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:11.329257 388 external_mini_cluster.cc:1428] Setting key 883d832eabf89365968d6e1fd7ff2287
I20260927 05:11:11.329665 20878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:11.344872 20878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d: Bootstrap starting.
Sep 27 05:11:11 dist-test-slave-hgs0 krb5kdc[20798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:11:11.348471 20878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:11.349654 20878 log.cc:824] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:11.352871 20878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d: No bootstrap required, opened a new log
I20260927 05:11:11.354993 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:11.331360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53137 (local address 127.0.0.1:33515)
0927 05:11:11.332244 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:11.332260 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:11.332307 (+ 47us) server_negotiation.cc:407] Connection header received
0927 05:11:11.334451 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:11.334477 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:11.334929 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:11.335331 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:11.337183 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:11.338534 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:11.340399 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:11.341192 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:11.341336 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:11.346502 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:11.346510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:11.346526 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:11.346573 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:11.350032 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:11.351004 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:11.351011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:11.351020 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:11.351208 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:11.351744 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:11.351748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:11.351750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:11.352383 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:11.352755 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:11.353352 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 05:11:11.353840 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":379,"threads_started":1}
I20260927 05:11:11.360132 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:11.362668 20878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f5bc1205a40e8aab2636d1841537d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33515 } }
I20260927 05:11:11.363145 20878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:11.363265 20878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419f5bc1205a40e8aab2636d1841537d, State: Initialized, Role: FOLLOWER
I20260927 05:11:11.364140 20878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f5bc1205a40e8aab2636d1841537d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33515 } }
I20260927 05:11:11.364408 20878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:11.364563 20878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:11.364749 20878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:11.367064 20878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f5bc1205a40e8aab2636d1841537d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33515 } }
I20260927 05:11:11.367690 20878 leader_election.cc:302] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 419f5bc1205a40e8aab2636d1841537d; no voters:
I20260927 05:11:11.368412 20878 leader_election.cc:288] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:11.368502 20887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:11.370683 20887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [term 1 LEADER]: Becoming Leader. State: Replica: 419f5bc1205a40e8aab2636d1841537d, State: Running, Role: LEADER
I20260927 05:11:11.371374 20887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f5bc1205a40e8aab2636d1841537d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33515 } }
I20260927 05:11:11.371791 20878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:11.374724 20888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "419f5bc1205a40e8aab2636d1841537d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f5bc1205a40e8aab2636d1841537d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33515 } } }
I20260927 05:11:11.375268 20888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:11.376197 20889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 419f5bc1205a40e8aab2636d1841537d. Latest consensus state: current_term: 1 leader_uuid: "419f5bc1205a40e8aab2636d1841537d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419f5bc1205a40e8aab2636d1841537d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33515 } } }
I20260927 05:11:11.376529 20889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:11.380545 20893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:11.388253 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:11.369171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35483 (local address 127.0.0.1:33515)
0927 05:11:11.369424 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:11.369432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:11.371658 (+ 2226us) server_negotiation.cc:407] Connection header received
0927 05:11:11.371892 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:11.371897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:11.372022 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:11.372148 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:11.374016 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:11.375279 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:11.377440 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:11.378432 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:11.378571 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:11.382024 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:11.382030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:11.382033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:11.382069 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:11.385063 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:11.385908 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:11.385914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:11.385918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:11.386059 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:11.386542 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:11.386547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:11.386552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:11.386906 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:11.387212 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:11.387622 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 05:11:11.387810 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:11:11.395201 20893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:11.405048 20893 catalog_manager.cc:1347] Generated new cluster ID: 04c68fb71ccc4801b7deeddacadfafbc
I20260927 05:11:11.405191 20893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:11.421626 20893 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:11.423651 20893 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:11.432868 20893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 419f5bc1205a40e8aab2636d1841537d: Generated new TSK 0
I20260927 05:11:11.434010 20893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:11.441537 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 20811
I20260927 05:11:11.442978 20873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:11.538992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 20811
W20260927 05:11:11.557282 20796 connection.cc:535] client connection to 127.0.0.1:33515 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33515): Connection reset by peer (error 104)
2026-09-27T05:11:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2147 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:11 dist-test-slave-hgs0 krb5kdc[20910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:11 dist-test-slave-hgs0 krb5kdc[20910](info): set up 2 sockets
Sep 27 05:11:11 dist-test-slave-hgs0 krb5kdc[20910](info): commencing operation
krb5kdc: starting...
W20260927 05:11:12.680300 388 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:12 dist-test-slave-hgs0 krb5kdc[20910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:12.953691 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:13.306210 20923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:13.306579 20923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:13.306744 20923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:13.314296 20923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:13.314469 20923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:13.314577 20923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:13.314697 20923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:13.314776 20923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:13.325105 20923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:13.326495 20923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:13.328024 20923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:13.340281 20928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:13.340205 20931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:13.340250 20929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:13.341576 20923 server_base.cc:1034] running on GCE node
I20260927 05:11:13.342474 20923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:13.343993 20923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:13.345239 20923 hybrid_clock.cc:648] HybridClock initialized: now 1790485873345201 us; error 41 us; skew 500 ppm
I20260927 05:11:13.345616 20923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:13 dist-test-slave-hgs0 krb5kdc[20910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:13.354511 20923 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:13.356484 20923 webserver.cc:460] Webserver started at http://127.0.0.1:45867/ using document root <none> and password file <none>
I20260927 05:11:13.357287 20923 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:13.357471 20923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:13.357829 20923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:13.361328 20923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "8f8d5106145d4a6eaefe38305d9eaa09"
format_stamp: "Formatted at 2026-09-27 05:11:13 on dist-test-slave-hgs0"
server_key: "faca5e5e87c8eeef18590052b22643da"
server_key_iv: "ed3410a894a9a692faaeb7cca729db9c"
server_key_version: "encryptionkey@0"
I20260927 05:11:13.362223 20923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f8d5106145d4a6eaefe38305d9eaa09"
format_stamp: "Formatted at 2026-09-27 05:11:13 on dist-test-slave-hgs0"
server_key: "faca5e5e87c8eeef18590052b22643da"
server_key_iv: "ed3410a894a9a692faaeb7cca729db9c"
server_key_version: "encryptionkey@0"
I20260927 05:11:13.367908 20923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:11:13.371644 20938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:13.373121 20923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:11:13.373342 20923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "8f8d5106145d4a6eaefe38305d9eaa09"
format_stamp: "Formatted at 2026-09-27 05:11:13 on dist-test-slave-hgs0"
server_key: "faca5e5e87c8eeef18590052b22643da"
server_key_iv: "ed3410a894a9a692faaeb7cca729db9c"
server_key_version: "encryptionkey@0"
I20260927 05:11:13.373600 20923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:13.397064 20923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:13.402555 20923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:13.402830 20923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:13.422371 20923 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42195
I20260927 05:11:13.422405 20989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42195 every 8 connection(s)
I20260927 05:11:13.423884 20923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:13.428056 20990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:13.429874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 20923
I20260927 05:11:13.430061 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:13.430485 388 external_mini_cluster.cc:1428] Setting key d0e07474ade2c4c532732a78980c69f0
I20260927 05:11:13.443594 20990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09: Bootstrap starting.
I20260927 05:11:13.446179 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:13.432533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58585 (local address 127.0.0.1:42195)
0927 05:11:13.433397 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:13.433414 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:13.433477 (+ 63us) server_negotiation.cc:407] Connection header received
0927 05:11:13.435296 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:13.435341 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:13.435999 (+ 658us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:13.436494 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:13.437011 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:13.438997 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:13.440567 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:13.441704 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:13.441889 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:13.443154 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:13.443165 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:13.443186 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:13.443987 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:13.444095 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:13.444126 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:13.444299 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:13.444573 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 05:11:13.444759 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":359,"threads_started":1}
I20260927 05:11:13.447924 20990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:13.449332 20990 log.cc:824] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:13.451068 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:13.452796 20990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09: No bootstrap required, opened a new log
I20260927 05:11:13.461498 20990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8d5106145d4a6eaefe38305d9eaa09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } }
I20260927 05:11:13.462081 20990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:13.462209 20990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8d5106145d4a6eaefe38305d9eaa09, State: Initialized, Role: FOLLOWER
I20260927 05:11:13.462975 20990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8d5106145d4a6eaefe38305d9eaa09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } }
I20260927 05:11:13.463315 20990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:13.463447 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:13.456019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58495 (local address 127.0.0.1:42195)
0927 05:11:13.456240 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:13.456249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:13.456763 (+ 514us) server_negotiation.cc:407] Connection header received
0927 05:11:13.457024 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:13.457028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:13.457099 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:13.457217 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:13.457843 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:13.458751 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:13.460459 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:13.460953 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:13.461105 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:13.462482 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:13.462488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:13.462491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:13.462512 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:13.462574 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:13.462591 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:13.462744 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:13.462974 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 05:11:13.463038 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:11:13.463508 20990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:13.463706 20990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:13.466831 20990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8d5106145d4a6eaefe38305d9eaa09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } }
I20260927 05:11:13.467471 20990 leader_election.cc:302] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f8d5106145d4a6eaefe38305d9eaa09; no voters:
I20260927 05:11:13.468266 20990 leader_election.cc:288] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:13.468377 21000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:13.469693 21000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [term 1 LEADER]: Becoming Leader. State: Replica: 8f8d5106145d4a6eaefe38305d9eaa09, State: Running, Role: LEADER
I20260927 05:11:13.470363 21000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8d5106145d4a6eaefe38305d9eaa09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } }
I20260927 05:11:13.471073 20990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:13.473588 21001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f8d5106145d4a6eaefe38305d9eaa09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8d5106145d4a6eaefe38305d9eaa09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } } }
I20260927 05:11:13.474078 21001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:13.474331 21002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f8d5106145d4a6eaefe38305d9eaa09. Latest consensus state: current_term: 1 leader_uuid: "8f8d5106145d4a6eaefe38305d9eaa09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f8d5106145d4a6eaefe38305d9eaa09" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42195 } } }
I20260927 05:11:13.474757 21002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:13.476246 21005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:13.483002 21005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:13.493229 21005 catalog_manager.cc:1347] Generated new cluster ID: 36d938a2f6f747f596fab0f1d20dba19
I20260927 05:11:13.493402 21005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:13.506443 21005 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:13.508412 21005 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:13.526090 21005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f8d5106145d4a6eaefe38305d9eaa09: Generated new TSK 0
I20260927 05:11:13.527467 21005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:13.544565 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 20923
W20260927 05:11:13.545948 20985 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58585
W20260927 05:11:13.547286 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 20923: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:13.547528 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 20923
W20260927 05:11:13.565466 20908 connection.cc:535] client connection to 127.0.0.1:42195 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42195): Connection reset by peer (error 104)
2026-09-27T05:11:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2008 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:13 dist-test-slave-hgs0 krb5kdc[21021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:13 dist-test-slave-hgs0 krb5kdc[21021](info): set up 2 sockets
Sep 27 05:11:13 dist-test-slave-hgs0 krb5kdc[21021](info): commencing operation
krb5kdc: starting...
W20260927 05:11:14.712289 388 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:14 dist-test-slave-hgs0 krb5kdc[21021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:14.989308 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:15.331151 21034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:15.331539 21034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:15.331714 21034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:15.339339 21034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:15.339510 21034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:15.339627 21034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:15.339740 21034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:15.339818 21034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:15.350054 21034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:15.351495 21034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:15.352957 21034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:15.365212 21039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:15.365303 21040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:15.365259 21042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:15.366346 21034 server_base.cc:1034] running on GCE node
I20260927 05:11:15.367162 21034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:15.368677 21034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:15.369889 21034 hybrid_clock.cc:648] HybridClock initialized: now 1790485875369826 us; error 56 us; skew 500 ppm
I20260927 05:11:15.370240 21034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:15 dist-test-slave-hgs0 krb5kdc[21021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:15.373553 21034 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:15.375078 21034 webserver.cc:460] Webserver started at http://127.0.0.1:39657/ using document root <none> and password file <none>
I20260927 05:11:15.375730 21034 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:15.375831 21034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:15.376140 21034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:15.379127 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "d06823730eef47119f8957c79f1cb044"
format_stamp: "Formatted at 2026-09-27 05:11:15 on dist-test-slave-hgs0"
server_key: "94fbc3f53758c0f4ba245825d71bf1f8"
server_key_iv: "719210ecb9f7c3ca1b76c0da2d5ea144"
server_key_version: "encryptionkey@0"
I20260927 05:11:15.379840 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "d06823730eef47119f8957c79f1cb044"
format_stamp: "Formatted at 2026-09-27 05:11:15 on dist-test-slave-hgs0"
server_key: "94fbc3f53758c0f4ba245825d71bf1f8"
server_key_iv: "719210ecb9f7c3ca1b76c0da2d5ea144"
server_key_version: "encryptionkey@0"
I20260927 05:11:15.385301 21034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:11:15.388959 21049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:15.390388 21034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 05:11:15.390585 21034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "d06823730eef47119f8957c79f1cb044"
format_stamp: "Formatted at 2026-09-27 05:11:15 on dist-test-slave-hgs0"
server_key: "94fbc3f53758c0f4ba245825d71bf1f8"
server_key_iv: "719210ecb9f7c3ca1b76c0da2d5ea144"
server_key_version: "encryptionkey@0"
I20260927 05:11:15.390779 21034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:15.396955 21034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:15.401304 21034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:15.401626 21034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:15.420127 21034 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43775
I20260927 05:11:15.420146 21100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43775 every 8 connection(s)
I20260927 05:11:15.421577 21034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:15.425351 21101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:15.425746 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 21034
I20260927 05:11:15.425926 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:15.426347 388 external_mini_cluster.cc:1428] Setting key bed1e9df1d72eade900e720ffd31dbd2
I20260927 05:11:15.434855 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:15.428118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43835 (local address 127.0.0.1:43775)
0927 05:11:15.428943 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:15.428954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:15.428995 (+ 41us) server_negotiation.cc:407] Connection header received
0927 05:11:15.431352 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:15.431377 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:15.431774 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:15.432210 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:15.433305 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:15.433312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:15.433325 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:15.433338 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:15.433396 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:15.433420 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:15.433582 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:15.433707 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 05:11:15.433911 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":389,"threads_started":1}
I20260927 05:11:15.438818 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:15.439388 21101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044: Bootstrap starting.
I20260927 05:11:15.442536 21101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:15.443971 21101 log.cc:824] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:15.445799 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:15.443256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43095 (local address 127.0.0.1:43775)
0927 05:11:15.443454 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:15.443462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:15.443767 (+ 305us) server_negotiation.cc:407] Connection header received
0927 05:11:15.444002 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:15.444008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:15.444071 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:15.444184 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:15.445152 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:15.445158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:15.445162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:15.445179 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:15.445226 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:15.445241 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:15.445359 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:15.445483 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 05:11:15.445542 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 05:11:15.446959 21101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044: No bootstrap required, opened a new log
I20260927 05:11:15.454420 21101 raft_consensus.cc:348] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06823730eef47119f8957c79f1cb044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43775 } }
I20260927 05:11:15.454795 21101 raft_consensus.cc:374] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:15.454869 21101 raft_consensus.cc:729] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06823730eef47119f8957c79f1cb044, State: Initialized, Role: FOLLOWER
I20260927 05:11:15.455446 21101 consensus_queue.cc:260] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06823730eef47119f8957c79f1cb044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43775 } }
I20260927 05:11:15.455679 21101 raft_consensus.cc:388] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:15.455840 21101 raft_consensus.cc:482] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:15.456079 21101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:15.458300 21101 raft_consensus.cc:504] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06823730eef47119f8957c79f1cb044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43775 } }
I20260927 05:11:15.458837 21101 leader_election.cc:302] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d06823730eef47119f8957c79f1cb044; no voters:
I20260927 05:11:15.459467 21101 leader_election.cc:288] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:15.459659 21111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:15.460458 21111 raft_consensus.cc:686] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [term 1 LEADER]: Becoming Leader. State: Replica: d06823730eef47119f8957c79f1cb044, State: Running, Role: LEADER
I20260927 05:11:15.461138 21111 consensus_queue.cc:237] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06823730eef47119f8957c79f1cb044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43775 } }
I20260927 05:11:15.461905 21101 sys_catalog.cc:564] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:15.464059 21113 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d06823730eef47119f8957c79f1cb044. Latest consensus state: current_term: 1 leader_uuid: "d06823730eef47119f8957c79f1cb044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06823730eef47119f8957c79f1cb044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43775 } } }
I20260927 05:11:15.464720 21113 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:15.464564 21112 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d06823730eef47119f8957c79f1cb044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06823730eef47119f8957c79f1cb044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43775 } } }
I20260927 05:11:15.464874 21112 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:15.468842 21119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:15.474110 21119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:15.483258 21119 catalog_manager.cc:1347] Generated new cluster ID: 6ab1e40fa0c947b7950239f8b38cd9de
I20260927 05:11:15.483403 21119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:15.503072 21119 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:15.503515 21119 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:15.514302 21119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d06823730eef47119f8957c79f1cb044: Generated new TSK 0
I20260927 05:11:15.515206 21119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:15.525170 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21034
W20260927 05:11:15.526369 21096 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43835
W20260927 05:11:15.527230 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21034: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:15.527395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 21034
2026-09-27T05:11:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1978 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:15 dist-test-slave-hgs0 krb5kdc[21132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:15 dist-test-slave-hgs0 krb5kdc[21132](info): set up 2 sockets
Sep 27 05:11:15 dist-test-slave-hgs0 krb5kdc[21132](info): commencing operation
krb5kdc: starting...
W20260927 05:11:16.644382 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:16 dist-test-slave-hgs0 krb5kdc[21132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:16.907050 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:17.270036 21145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:17.270391 21145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:17.270552 21145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:17.278568 21145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:17.278715 21145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:17.278810 21145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:17.278934 21145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:17.279017 21145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:17.290299 21145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:17.291808 21145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:17.293359 21145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:17.307035 21150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:17.307178 21151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:17.307852 21153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:17.308429 21145 server_base.cc:1034] running on GCE node
I20260927 05:11:17.309469 21145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:17.311091 21145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:17.312338 21145 hybrid_clock.cc:648] HybridClock initialized: now 1790485877312259 us; error 64 us; skew 500 ppm
I20260927 05:11:17.312819 21145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:17 dist-test-slave-hgs0 krb5kdc[21132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:17.316394 21145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:17.318095 21145 webserver.cc:460] Webserver started at http://127.0.0.1:33349/ using document root <none> and password file <none>
I20260927 05:11:17.318892 21145 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:17.319058 21145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:17.319397 21145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:17.322937 21145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "66b85dd3cdb24d8ca1a8d09d334193b3"
format_stamp: "Formatted at 2026-09-27 05:11:17 on dist-test-slave-hgs0"
server_key: "7d0c2b4a76f376e0e175083b2d7746e8"
server_key_iv: "df9a7f00cb0e94a259ada3e4ef1cd50d"
server_key_version: "encryptionkey@0"
I20260927 05:11:17.323814 21145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "66b85dd3cdb24d8ca1a8d09d334193b3"
format_stamp: "Formatted at 2026-09-27 05:11:17 on dist-test-slave-hgs0"
server_key: "7d0c2b4a76f376e0e175083b2d7746e8"
server_key_iv: "df9a7f00cb0e94a259ada3e4ef1cd50d"
server_key_version: "encryptionkey@0"
I20260927 05:11:17.330266 21145 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:11:17.334656 21160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:17.336506 21145 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 05:11:17.336752 21145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "66b85dd3cdb24d8ca1a8d09d334193b3"
format_stamp: "Formatted at 2026-09-27 05:11:17 on dist-test-slave-hgs0"
server_key: "7d0c2b4a76f376e0e175083b2d7746e8"
server_key_iv: "df9a7f00cb0e94a259ada3e4ef1cd50d"
server_key_version: "encryptionkey@0"
I20260927 05:11:17.337002 21145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:17.350497 21145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:17.355376 21145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:17.355696 21145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:17.375734 21145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39411
I20260927 05:11:17.375756 21211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39411 every 8 connection(s)
I20260927 05:11:17.377306 21145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:17.381199 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:17.384275 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 21145
I20260927 05:11:17.384469 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:17.384984 388 external_mini_cluster.cc:1428] Setting key 572601605cd95ccacb5f2211075d6cc2
I20260927 05:11:17.393605 21212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3: Bootstrap starting.
I20260927 05:11:17.397202 21212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:17.398309 21212 log.cc:824] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:11:17 dist-test-slave-hgs0 krb5kdc[21132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:11:17.401724 21212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3: No bootstrap required, opened a new log
I20260927 05:11:17.409498 21215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:17.386954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34363 (local address 127.0.0.1:39411)
0927 05:11:17.387785 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:17.387799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:17.387849 (+ 50us) server_negotiation.cc:407] Connection header received
0927 05:11:17.389636 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:17.389662 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:17.390087 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:17.390570 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:17.392184 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:17.393555 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:17.394958 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:17.395730 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:17.395952 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:17.400904 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:17.400914 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:17.400931 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:17.400973 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:17.404577 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:17.405573 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:17.405581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:17.405589 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:17.405812 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:17.406267 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:17.406275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:17.406277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:17.406961 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:17.407211 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:17.407721 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 05:11:17.408323 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":359,"threads_started":1}
I20260927 05:11:17.412396 21212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b85dd3cdb24d8ca1a8d09d334193b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39411 } }
I20260927 05:11:17.412876 21212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:17.413009 21212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b85dd3cdb24d8ca1a8d09d334193b3, State: Initialized, Role: FOLLOWER
I20260927 05:11:17.413805 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:17.413733 21212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b85dd3cdb24d8ca1a8d09d334193b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39411 } }
I20260927 05:11:17.414074 21212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:17.414242 21212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:17.414438 21212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:17.417001 21212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b85dd3cdb24d8ca1a8d09d334193b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39411 } }
I20260927 05:11:17.417758 21212 leader_election.cc:302] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b85dd3cdb24d8ca1a8d09d334193b3; no voters:
I20260927 05:11:17.418433 21212 leader_election.cc:288] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:17.418599 21221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:17.420970 21212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:17.425596 21221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [term 1 LEADER]: Becoming Leader. State: Replica: 66b85dd3cdb24d8ca1a8d09d334193b3, State: Running, Role: LEADER
I20260927 05:11:17.426256 21221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b85dd3cdb24d8ca1a8d09d334193b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39411 } }
I20260927 05:11:17.429281 21222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b85dd3cdb24d8ca1a8d09d334193b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b85dd3cdb24d8ca1a8d09d334193b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39411 } } }
I20260927 05:11:17.429344 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b85dd3cdb24d8ca1a8d09d334193b3. Latest consensus state: current_term: 1 leader_uuid: "66b85dd3cdb24d8ca1a8d09d334193b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b85dd3cdb24d8ca1a8d09d334193b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39411 } } }
I20260927 05:11:17.429723 21222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:17.429751 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:17.431077 21229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:17.441565 21215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:17.419707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33717 (local address 127.0.0.1:39411)
0927 05:11:17.419889 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:17.419897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:17.420291 (+ 394us) server_negotiation.cc:407] Connection header received
0927 05:11:17.420760 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:17.420765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:17.420853 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:17.420977 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:17.423514 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:17.425057 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:17.426842 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:17.427806 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:17.428008 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:17.431483 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:17.431489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:17.431494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:17.431533 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:17.436209 (+ 4676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:17.437428 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:17.437437 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:17.437443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:17.437635 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:17.438156 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:17.438160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:17.438163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:17.438655 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:17.438978 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:17.440673 (+ 1695us) server_negotiation.cc:299] Negotiation successful
0927 05:11:17.441020 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:11:17.444517 21229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:17.453500 21229 catalog_manager.cc:1347] Generated new cluster ID: d141d60b62d744008746c37802bf621d
I20260927 05:11:17.453637 21229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:17.478441 21229 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:17.480455 21229 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:17.497346 21229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66b85dd3cdb24d8ca1a8d09d334193b3: Generated new TSK 0
I20260927 05:11:17.498673 21229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:17.506678 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21145
I20260927 05:11:17.507983 21207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:17.603358 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 21145
2026-09-27T05:11:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:17 dist-test-slave-hgs0 krb5kdc[21244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:17 dist-test-slave-hgs0 krb5kdc[21244](info): set up 2 sockets
Sep 27 05:11:17 dist-test-slave-hgs0 krb5kdc[21244](info): commencing operation
krb5kdc: starting...
W20260927 05:11:18.720402 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:18 dist-test-slave-hgs0 krb5kdc[21244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:18.990233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:19.343993 21257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:19.344377 21257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:19.344566 21257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:19.352034 21257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:19.352208 21257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:19.352315 21257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:19.352411 21257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:19.352478 21257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:19.362764 21257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:19.364225 21257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:19.365612 21257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:19.378362 21262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:19.378329 21263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:19.378298 21265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:19.379148 21257 server_base.cc:1034] running on GCE node
I20260927 05:11:19.380100 21257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:19.381574 21257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:19.382823 21257 hybrid_clock.cc:648] HybridClock initialized: now 1790485879382773 us; error 47 us; skew 500 ppm
I20260927 05:11:19.383198 21257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:19 dist-test-slave-hgs0 krb5kdc[21244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:19.393780 21257 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:19.395428 21257 webserver.cc:460] Webserver started at http://127.0.0.1:38371/ using document root <none> and password file <none>
I20260927 05:11:19.396262 21257 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:19.396423 21257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:19.396775 21257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:19.400045 21257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "5ec5586deb074268910385f87d93422f"
format_stamp: "Formatted at 2026-09-27 05:11:19 on dist-test-slave-hgs0"
server_key: "ff84ba959dc8c7f7b3ba16bc062952b6"
server_key_iv: "aa1374427d448eefc548de799d6c8814"
server_key_version: "encryptionkey@0"
I20260927 05:11:19.400856 21257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ec5586deb074268910385f87d93422f"
format_stamp: "Formatted at 2026-09-27 05:11:19 on dist-test-slave-hgs0"
server_key: "ff84ba959dc8c7f7b3ba16bc062952b6"
server_key_iv: "aa1374427d448eefc548de799d6c8814"
server_key_version: "encryptionkey@0"
I20260927 05:11:19.406352 21257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:11:19.410096 21272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:19.411506 21257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:11:19.411700 21257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "5ec5586deb074268910385f87d93422f"
format_stamp: "Formatted at 2026-09-27 05:11:19 on dist-test-slave-hgs0"
server_key: "ff84ba959dc8c7f7b3ba16bc062952b6"
server_key_iv: "aa1374427d448eefc548de799d6c8814"
server_key_version: "encryptionkey@0"
I20260927 05:11:19.411909 21257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:19.439709 21257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:19.444247 21257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:19.444574 21257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:19.463510 21257 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43727
I20260927 05:11:19.463528 21323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43727 every 8 connection(s)
I20260927 05:11:19.465003 21257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:19.468719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 21257
I20260927 05:11:19.468940 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:19.468993 21324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:19.469378 388 external_mini_cluster.cc:1428] Setting key d5ae90bfb7e2eddd99903c962c03789c
I20260927 05:11:19.485039 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f: Bootstrap starting.
I20260927 05:11:19.485121 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:19.471233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59941 (local address 127.0.0.1:43727)
0927 05:11:19.472118 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:19.472135 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:19.472191 (+ 56us) server_negotiation.cc:407] Connection header received
0927 05:11:19.474211 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:19.474242 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:19.474829 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:19.475330 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:19.475901 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:19.478054 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:19.479456 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:19.480588 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:19.480789 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:19.482098 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:19.482110 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:19.482132 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:19.482940 (+ 808us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:19.483039 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:19.483076 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:19.483177 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:19.483428 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 05:11:19.483719 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":353,"threads_started":1}
I20260927 05:11:19.488665 21324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:19.489449 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:19.489894 21324 log.cc:824] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:19.493201 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f: No bootstrap required, opened a new log
I20260927 05:11:19.501543 21327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:19.494429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43957 (local address 127.0.0.1:43727)
0927 05:11:19.494626 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:19.494634 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:19.495013 (+ 379us) server_negotiation.cc:407] Connection header received
0927 05:11:19.495169 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:19.495174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:19.495253 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:19.495374 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:19.495889 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:19.497064 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:19.498482 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:19.499221 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:19.499363 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:19.500546 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:19.500559 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:19.500564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:19.500590 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:19.500644 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:19.500666 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:19.500782 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:19.501025 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 05:11:19.501145 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 05:11:19.503165 21324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec5586deb074268910385f87d93422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } }
I20260927 05:11:19.503710 21324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:19.503799 21324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec5586deb074268910385f87d93422f, State: Initialized, Role: FOLLOWER
I20260927 05:11:19.504374 21324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec5586deb074268910385f87d93422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } }
I20260927 05:11:19.504608 21324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:19.504724 21324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:19.504854 21324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:19.507421 21324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec5586deb074268910385f87d93422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } }
I20260927 05:11:19.508137 21324 leader_election.cc:302] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ec5586deb074268910385f87d93422f; no voters:
I20260927 05:11:19.508816 21324 leader_election.cc:288] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:19.509008 21334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:19.509820 21334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [term 1 LEADER]: Becoming Leader. State: Replica: 5ec5586deb074268910385f87d93422f, State: Running, Role: LEADER
I20260927 05:11:19.510452 21334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec5586deb074268910385f87d93422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } }
I20260927 05:11:19.511126 21324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:19.514132 21336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ec5586deb074268910385f87d93422f. Latest consensus state: current_term: 1 leader_uuid: "5ec5586deb074268910385f87d93422f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec5586deb074268910385f87d93422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } } }
I20260927 05:11:19.514591 21336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:19.515493 21335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ec5586deb074268910385f87d93422f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec5586deb074268910385f87d93422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43727 } } }
I20260927 05:11:19.516072 21335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:19.517201 21342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:19.523458 21342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:19.533802 21342 catalog_manager.cc:1347] Generated new cluster ID: 00f68f92ea8f480bbaa679f26ba1595b
I20260927 05:11:19.533980 21342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:19.550854 21342 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:19.552387 21342 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:19.566805 21342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ec5586deb074268910385f87d93422f: Generated new TSK 0
I20260927 05:11:19.567746 21342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:19.575278 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21257
W20260927 05:11:19.576848 21319 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59941
W20260927 05:11:19.577695 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21257: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:19.577868 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 21257
2026-09-27T05:11:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:19 dist-test-slave-hgs0 krb5kdc[21355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:19 dist-test-slave-hgs0 krb5kdc[21355](info): set up 2 sockets
Sep 27 05:11:19 dist-test-slave-hgs0 krb5kdc[21355](info): commencing operation
krb5kdc: starting...
W20260927 05:11:20.720853 388 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.012s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:20 dist-test-slave-hgs0 krb5kdc[21355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:20.997529 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:21.356909 21368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:21.357226 21368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:21.357401 21368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:21.364959 21368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:21.365089 21368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:21.365180 21368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:21.365284 21368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:21.365377 21368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:21.375639 21368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:21.377084 21368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:21.378424 21368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:21.391158 21376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:21.391193 21373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:21.391117 21374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:21.392040 21368 server_base.cc:1034] running on GCE node
I20260927 05:11:21.392943 21368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:21.394519 21368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:21.395767 21368 hybrid_clock.cc:648] HybridClock initialized: now 1790485881395726 us; error 30 us; skew 500 ppm
I20260927 05:11:21.396216 21368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:21 dist-test-slave-hgs0 krb5kdc[21355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:21.399454 21368 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:21.401170 21368 webserver.cc:460] Webserver started at http://127.0.0.1:36323/ using document root <none> and password file <none>
I20260927 05:11:21.401921 21368 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:21.402074 21368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:21.402401 21368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:21.405678 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9"
format_stamp: "Formatted at 2026-09-27 05:11:21 on dist-test-slave-hgs0"
server_key: "a2a067d9ecded1852ed8f580c4a2210a"
server_key_iv: "e68052e18927edb39d8947490714e3b5"
server_key_version: "encryptionkey@0"
I20260927 05:11:21.406533 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9"
format_stamp: "Formatted at 2026-09-27 05:11:21 on dist-test-slave-hgs0"
server_key: "a2a067d9ecded1852ed8f580c4a2210a"
server_key_iv: "e68052e18927edb39d8947490714e3b5"
server_key_version: "encryptionkey@0"
I20260927 05:11:21.412743 21368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 05:11:21.417395 21383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:21.419438 21368 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 05:11:21.419727 21368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9"
format_stamp: "Formatted at 2026-09-27 05:11:21 on dist-test-slave-hgs0"
server_key: "a2a067d9ecded1852ed8f580c4a2210a"
server_key_iv: "e68052e18927edb39d8947490714e3b5"
server_key_version: "encryptionkey@0"
I20260927 05:11:21.420082 21368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:21.427295 21368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:21.432176 21368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:21.432520 21368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:21.453362 21368 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46655
I20260927 05:11:21.453408 21434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46655 every 8 connection(s)
I20260927 05:11:21.454932 21368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:21.458117 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 21368
I20260927 05:11:21.458317 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:21.458851 388 external_mini_cluster.cc:1428] Setting key 888a4df3c6f4fbaf04f2dfaaee880b20
I20260927 05:11:21.459581 21435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:21.467192 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:21.460810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60795 (local address 127.0.0.1:46655)
0927 05:11:21.461666 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:21.461679 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:21.461724 (+ 45us) server_negotiation.cc:407] Connection header received
0927 05:11:21.463369 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:21.463393 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:21.463808 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:21.464252 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:21.465438 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:21.465446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:21.465460 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:21.465475 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:21.465558 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:21.465580 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:21.465764 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:21.465928 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 05:11:21.466174 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":449,"threads_started":1}
I20260927 05:11:21.471483 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:21.474431 21435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9: Bootstrap starting.
I20260927 05:11:21.477967 21435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:21.478821 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:21.476034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54607 (local address 127.0.0.1:46655)
0927 05:11:21.476293 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:21.476304 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:21.476823 (+ 519us) server_negotiation.cc:407] Connection header received
0927 05:11:21.477035 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:21.477040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:21.477104 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:21.477218 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:21.478172 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:21.478178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:21.478183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:21.478201 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:21.478253 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:21.478269 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:21.478416 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:21.478478 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 05:11:21.478542 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:11:21.479347 21435 log.cc:824] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:21.482391 21435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9: No bootstrap required, opened a new log
I20260927 05:11:21.491142 21435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46655 } }
I20260927 05:11:21.491678 21435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:21.491801 21435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7412fd4ca14c4416b6c3a7b84ce74ff9, State: Initialized, Role: FOLLOWER
I20260927 05:11:21.492687 21435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46655 } }
I20260927 05:11:21.493003 21435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:21.493170 21435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:21.493404 21435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:21.496543 21435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46655 } }
I20260927 05:11:21.497354 21435 leader_election.cc:302] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7412fd4ca14c4416b6c3a7b84ce74ff9; no voters:
I20260927 05:11:21.498220 21435 leader_election.cc:288] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:21.498464 21445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:21.499415 21445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 7412fd4ca14c4416b6c3a7b84ce74ff9, State: Running, Role: LEADER
I20260927 05:11:21.500223 21445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46655 } }
I20260927 05:11:21.501078 21435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:21.503727 21447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7412fd4ca14c4416b6c3a7b84ce74ff9. Latest consensus state: current_term: 1 leader_uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46655 } } }
I20260927 05:11:21.503995 21446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7412fd4ca14c4416b6c3a7b84ce74ff9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46655 } } }
I20260927 05:11:21.504316 21447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:21.504355 21446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:21.507184 21452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:21.513289 21452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:21.523631 21452 catalog_manager.cc:1347] Generated new cluster ID: 9abf669dfcd34bcf8a78eac16b761b86
I20260927 05:11:21.523782 21452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:21.542582 21452 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:21.543098 21452 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:21.552819 21452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7412fd4ca14c4416b6c3a7b84ce74ff9: Generated new TSK 0
I20260927 05:11:21.553957 21452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:21.566862 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21368
W20260927 05:11:21.568259 21430 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60795
W20260927 05:11:21.569097 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:21.569271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 21368
2026-09-27T05:11:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1992 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:21 dist-test-slave-hgs0 krb5kdc[21466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:21 dist-test-slave-hgs0 krb5kdc[21466](info): set up 2 sockets
Sep 27 05:11:21 dist-test-slave-hgs0 krb5kdc[21466](info): commencing operation
krb5kdc: starting...
W20260927 05:11:22.720317 388 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:22 dist-test-slave-hgs0 krb5kdc[21466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:22.992959 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:23.336431 21479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:23.336742 21479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:23.336833 21479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:23.344242 21479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:23.344362 21479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:23.344408 21479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:23.344471 21479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:23.344511 21479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:23.354568 21479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:23.355901 21479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:23.357268 21479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:23.369966 21484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:23.370008 21485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:23.369932 21487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:23.370900 21479 server_base.cc:1034] running on GCE node
I20260927 05:11:23.371785 21479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:23.373382 21479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:23.374588 21479 hybrid_clock.cc:648] HybridClock initialized: now 1790485883374522 us; error 60 us; skew 500 ppm
I20260927 05:11:23.374998 21479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:23 dist-test-slave-hgs0 krb5kdc[21466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:23.378320 21479 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:23.380046 21479 webserver.cc:460] Webserver started at http://127.0.0.1:33017/ using document root <none> and password file <none>
I20260927 05:11:23.380784 21479 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:23.380931 21479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:23.381264 21479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:23.384441 21479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "ffd8877156f14afca3411ce054278789"
format_stamp: "Formatted at 2026-09-27 05:11:23 on dist-test-slave-hgs0"
server_key: "f96cd7285b576205c3008832b10661dd"
server_key_iv: "b1042d2e8c6fa022bd2a423b22005f93"
server_key_version: "encryptionkey@0"
I20260927 05:11:23.385262 21479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "ffd8877156f14afca3411ce054278789"
format_stamp: "Formatted at 2026-09-27 05:11:23 on dist-test-slave-hgs0"
server_key: "f96cd7285b576205c3008832b10661dd"
server_key_iv: "b1042d2e8c6fa022bd2a423b22005f93"
server_key_version: "encryptionkey@0"
I20260927 05:11:23.390933 21479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:11:23.394830 21494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:23.396425 21479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:11:23.396641 21479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "ffd8877156f14afca3411ce054278789"
format_stamp: "Formatted at 2026-09-27 05:11:23 on dist-test-slave-hgs0"
server_key: "f96cd7285b576205c3008832b10661dd"
server_key_iv: "b1042d2e8c6fa022bd2a423b22005f93"
server_key_version: "encryptionkey@0"
I20260927 05:11:23.396868 21479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:23.426301 21479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:23.431208 21479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:23.431561 21479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:23.452603 21479 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41735
I20260927 05:11:23.452634 21545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41735 every 8 connection(s)
I20260927 05:11:23.454090 21479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:23.458364 21546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:23.462368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 21479
I20260927 05:11:23.462572 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:23.463155 388 external_mini_cluster.cc:1428] Setting key d346fd02717d482fe92aa2189b2c4bf7
I20260927 05:11:23.472373 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789: Bootstrap starting.
I20260927 05:11:23.476689 21546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:23.478255 21546 log.cc:824] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789: Log is configured to *not* fsync() on all Append() calls
Sep 27 05:11:23 dist-test-slave-hgs0 krb5kdc[21466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:11:23.482228 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789: No bootstrap required, opened a new log
I20260927 05:11:23.490151 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:23.465235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:58951 (local address 127.0.0.1:41735)
0927 05:11:23.466068 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:23.466081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:23.466123 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:11:23.467827 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:23.467851 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:23.468407 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:23.468784 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:23.470574 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:23.471999 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:23.473888 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:23.474694 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:23.474839 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:23.480755 (+ 5916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:23.480772 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:23.480792 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:23.480840 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:23.484851 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:23.485908 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:23.485917 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:23.485925 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:23.486142 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:23.486808 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:23.486814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:23.486817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:23.487462 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:23.487884 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:23.488500 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 05:11:23.488892 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":385,"threads_started":1}
I20260927 05:11:23.492883 21546 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd8877156f14afca3411ce054278789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260927 05:11:23.493408 21546 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:23.493530 21546 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd8877156f14afca3411ce054278789, State: Initialized, Role: FOLLOWER
I20260927 05:11:23.494200 21546 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd8877156f14afca3411ce054278789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260927 05:11:23.494483 21546 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:23.494656 21546 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:23.494853 21546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:23.495167 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:23.497659 21546 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd8877156f14afca3411ce054278789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260927 05:11:23.498418 21546 leader_election.cc:302] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd8877156f14afca3411ce054278789; no voters:
I20260927 05:11:23.499156 21546 leader_election.cc:288] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:23.499270 21555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:23.499922 21555 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [term 1 LEADER]: Becoming Leader. State: Replica: ffd8877156f14afca3411ce054278789, State: Running, Role: LEADER
I20260927 05:11:23.500837 21555 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd8877156f14afca3411ce054278789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260927 05:11:23.503089 21546 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:23.503360 21555 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffd8877156f14afca3411ce054278789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd8877156f14afca3411ce054278789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } } }
I20260927 05:11:23.503664 21555 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:23.506625 21560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:23.506425 21557 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd8877156f14afca3411ce054278789. Latest consensus state: current_term: 1 leader_uuid: "ffd8877156f14afca3411ce054278789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd8877156f14afca3411ce054278789" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } } }
I20260927 05:11:23.506851 21557 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:23.518703 21560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:23.524403 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:23.500683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:33609 (local address 127.0.0.1:41735)
0927 05:11:23.500909 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:23.500918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:23.501186 (+ 268us) server_negotiation.cc:407] Connection header received
0927 05:11:23.501435 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:23.501459 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:23.501561 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:23.501642 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:23.504180 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:23.506202 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:23.510509 (+ 4307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:23.511508 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:23.511683 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:23.515991 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:23.515999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:23.516004 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:23.516047 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:23.520170 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:23.521304 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:23.521310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:23.521313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:23.521502 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:23.522085 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:23.522091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:23.522095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:23.522572 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:23.522831 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:23.523487 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 05:11:23.523677 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:11:23.531358 21560 catalog_manager.cc:1347] Generated new cluster ID: 02bc748c59b64b1a840ee09faa2cfd19
I20260927 05:11:23.531507 21560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:23.544021 21560 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:23.545773 21560 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:23.551561 21560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffd8877156f14afca3411ce054278789: Generated new TSK 0
I20260927 05:11:23.552605 21560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:23.561278 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21479
I20260927 05:11:23.562635 21541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:23.654306 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 21479
W20260927 05:11:23.671414 21464 connection.cc:535] client connection to 127.0.0.1:41735 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41735): Connection reset by peer (error 104)
2026-09-27T05:11:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2085 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:23 dist-test-slave-hgs0 krb5kdc[21578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:23 dist-test-slave-hgs0 krb5kdc[21578](info): set up 2 sockets
Sep 27 05:11:23 dist-test-slave-hgs0 krb5kdc[21578](info): commencing operation
krb5kdc: starting...
W20260927 05:11:24.772315 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:24 dist-test-slave-hgs0 krb5kdc[21578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:25.052333 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:25.407559 21591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:25.407970 21591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:25.408126 21591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:25.415691 21591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:25.415819 21591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:25.415925 21591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:25.416047 21591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:25.416138 21591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:25.426539 21591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:25.428114 21591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:25.429621 21591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:25.443423 21596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:25.443470 21597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:25.443393 21599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:25.444414 21591 server_base.cc:1034] running on GCE node
I20260927 05:11:25.445271 21591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:25.446875 21591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:25.448112 21591 hybrid_clock.cc:648] HybridClock initialized: now 1790485885448059 us; error 49 us; skew 500 ppm
I20260927 05:11:25.448488 21591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:25 dist-test-slave-hgs0 krb5kdc[21578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:25.461534 21591 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:25.463296 21591 webserver.cc:460] Webserver started at http://127.0.0.1:46325/ using document root <none> and password file <none>
I20260927 05:11:25.464141 21591 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:25.464291 21591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:25.464612 21591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:25.467650 21591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "5aea785b4e7f4fe0bb6e74b24de9df50"
format_stamp: "Formatted at 2026-09-27 05:11:25 on dist-test-slave-hgs0"
server_key: "859ea18ae33f0641223d1d2afe23c078"
server_key_iv: "3cd27a3a0dd9b0c730afa82154107cb9"
server_key_version: "encryptionkey@0"
I20260927 05:11:25.468485 21591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "5aea785b4e7f4fe0bb6e74b24de9df50"
format_stamp: "Formatted at 2026-09-27 05:11:25 on dist-test-slave-hgs0"
server_key: "859ea18ae33f0641223d1d2afe23c078"
server_key_iv: "3cd27a3a0dd9b0c730afa82154107cb9"
server_key_version: "encryptionkey@0"
I20260927 05:11:25.473824 21591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:11:25.477588 21606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:25.479067 21591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 05:11:25.479259 21591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "5aea785b4e7f4fe0bb6e74b24de9df50"
format_stamp: "Formatted at 2026-09-27 05:11:25 on dist-test-slave-hgs0"
server_key: "859ea18ae33f0641223d1d2afe23c078"
server_key_iv: "3cd27a3a0dd9b0c730afa82154107cb9"
server_key_version: "encryptionkey@0"
I20260927 05:11:25.479473 21591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:25.496083 21591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:25.500674 21591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:25.500988 21591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:25.520484 21591 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38137
I20260927 05:11:25.520529 21657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38137 every 8 connection(s)
I20260927 05:11:25.522101 21591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:25.526204 21658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:25.529816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 21591
I20260927 05:11:25.530009 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:25.530596 388 external_mini_cluster.cc:1428] Setting key afb48ba0c9152c6b08173700d409ea52
I20260927 05:11:25.540035 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50: Bootstrap starting.
I20260927 05:11:25.543964 21658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:25.545209 21658 log.cc:824] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:25.545761 21661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:25.532792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:40011 (local address 127.0.0.1:38137)
0927 05:11:25.533678 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:25.533691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:25.533737 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:11:25.535618 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:25.535646 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:25.536280 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:25.536701 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:25.537322 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:25.538780 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:25.540913 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:25.541643 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:25.541813 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:25.543129 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:25.543137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:25.543151 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:25.543703 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:25.543779 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:25.543808 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:25.544017 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:25.544402 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 05:11:25.544604 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":396,"threads_started":1}
I20260927 05:11:25.548308 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50: No bootstrap required, opened a new log
I20260927 05:11:25.549794 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:25.556972 21658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aea785b4e7f4fe0bb6e74b24de9df50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260927 05:11:25.557471 21658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:25.557588 21658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aea785b4e7f4fe0bb6e74b24de9df50, State: Initialized, Role: FOLLOWER
I20260927 05:11:25.558269 21658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aea785b4e7f4fe0bb6e74b24de9df50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260927 05:11:25.558588 21658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:25.558748 21658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:25.558938 21658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:25.561051 21661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:25.554236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:44795 (local address 127.0.0.1:38137)
0927 05:11:25.554451 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:25.554459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:25.554764 (+ 305us) server_negotiation.cc:407] Connection header received
0927 05:11:25.554973 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:25.554976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:25.555044 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:25.555128 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:25.555699 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:25.556516 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:25.558361 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:25.558972 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:25.559104 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:25.560309 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:25.560316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:25.560325 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:25.560353 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:25.560403 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:25.560418 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:25.560537 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:25.560725 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 05:11:25.560777 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 05:11:25.561530 21658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aea785b4e7f4fe0bb6e74b24de9df50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260927 05:11:25.562040 21658 leader_election.cc:302] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aea785b4e7f4fe0bb6e74b24de9df50; no voters:
I20260927 05:11:25.562687 21658 leader_election.cc:288] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:25.562752 21668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:25.563539 21668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [term 1 LEADER]: Becoming Leader. State: Replica: 5aea785b4e7f4fe0bb6e74b24de9df50, State: Running, Role: LEADER
I20260927 05:11:25.564163 21668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aea785b4e7f4fe0bb6e74b24de9df50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260927 05:11:25.564879 21658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:25.567092 21670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aea785b4e7f4fe0bb6e74b24de9df50. Latest consensus state: current_term: 1 leader_uuid: "5aea785b4e7f4fe0bb6e74b24de9df50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aea785b4e7f4fe0bb6e74b24de9df50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } } }
I20260927 05:11:25.567467 21670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:25.567067 21669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aea785b4e7f4fe0bb6e74b24de9df50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aea785b4e7f4fe0bb6e74b24de9df50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } } }
I20260927 05:11:25.572350 21669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:25.573046 21675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:25.578141 21675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:25.588294 21675 catalog_manager.cc:1347] Generated new cluster ID: 245529dc9ec140c4aa0ffc0dde100237
I20260927 05:11:25.588423 21675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:25.624188 21675 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:25.626168 21675 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:25.658493 21675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aea785b4e7f4fe0bb6e74b24de9df50: Generated new TSK 0
I20260927 05:11:25.659572 21675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:25.676800 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21591
W20260927 05:11:25.677997 21653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:40011
W20260927 05:11:25.678830 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:25.678982 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 21591
W20260927 05:11:25.696511 21576 connection.cc:535] client connection to 127.0.0.1:38137 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38137): Connection reset by peer (error 104)
2026-09-27T05:11:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:25 dist-test-slave-hgs0 krb5kdc[21689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:25 dist-test-slave-hgs0 krb5kdc[21689](info): set up 2 sockets
Sep 27 05:11:25 dist-test-slave-hgs0 krb5kdc[21689](info): commencing operation
krb5kdc: starting...
W20260927 05:11:26.796705 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:26 dist-test-slave-hgs0 krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:27.067324 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:27.405655 21702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:27.405941 21702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:27.406105 21702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:27.413316 21702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:27.413439 21702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:27.413530 21702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:27.413616 21702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:27.413707 21702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:27.423786 21702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:27.425171 21702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:27.426523 21702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:27.438552 21707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:27.438638 21708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:27.438598 21710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:27.439779 21702 server_base.cc:1034] running on GCE node
I20260927 05:11:27.440632 21702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:27.442090 21702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:27.443312 21702 hybrid_clock.cc:648] HybridClock initialized: now 1790485887443259 us; error 53 us; skew 500 ppm
I20260927 05:11:27.443665 21702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:27 dist-test-slave-hgs0 krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:27.446853 21702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:27.448415 21702 webserver.cc:460] Webserver started at http://127.0.0.1:46267/ using document root <none> and password file <none>
I20260927 05:11:27.449057 21702 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:27.449156 21702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:27.449420 21702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:27.452454 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "2544211de6b14b8c83e543473d3e4695"
format_stamp: "Formatted at 2026-09-27 05:11:27 on dist-test-slave-hgs0"
server_key: "d870930d5666d6f80b3ee51d04f17ce8"
server_key_iv: "e327b4b491506527fd9f4ca597addc05"
server_key_version: "encryptionkey@0"
I20260927 05:11:27.453171 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "2544211de6b14b8c83e543473d3e4695"
format_stamp: "Formatted at 2026-09-27 05:11:27 on dist-test-slave-hgs0"
server_key: "d870930d5666d6f80b3ee51d04f17ce8"
server_key_iv: "e327b4b491506527fd9f4ca597addc05"
server_key_version: "encryptionkey@0"
I20260927 05:11:27.458778 21702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:11:27.462687 21717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:27.464275 21702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 05:11:27.464430 21702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "2544211de6b14b8c83e543473d3e4695"
format_stamp: "Formatted at 2026-09-27 05:11:27 on dist-test-slave-hgs0"
server_key: "d870930d5666d6f80b3ee51d04f17ce8"
server_key_iv: "e327b4b491506527fd9f4ca597addc05"
server_key_version: "encryptionkey@0"
I20260927 05:11:27.464609 21702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:27.470505 21702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:27.474828 21702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:27.475092 21702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:27.493892 21702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40247
I20260927 05:11:27.493933 21768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40247 every 8 connection(s)
I20260927 05:11:27.495383 21702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:27.499429 21769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:27.502616 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 21702
I20260927 05:11:27.502823 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:27.503365 388 external_mini_cluster.cc:1428] Setting key f25ab9277c4cfcd22114cf372edb56c2
I20260927 05:11:27.511754 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:27.505368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:45037 (local address 127.0.0.1:40247)
0927 05:11:27.506180 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:27.506197 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:27.506241 (+ 44us) server_negotiation.cc:407] Connection header received
0927 05:11:27.508026 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:27.508060 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:27.508438 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:27.508886 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:27.510064 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:27.510071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:27.510085 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:27.510100 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:27.510163 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:27.510183 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:27.510450 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:27.510597 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 05:11:27.510793 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":339,"threads_started":1}
I20260927 05:11:27.512691 21769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695: Bootstrap starting.
I20260927 05:11:27.515698 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:27.516391 21769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:27.517433 21769 log.cc:824] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:27.520854 21769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695: No bootstrap required, opened a new log
I20260927 05:11:27.522809 21772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:27.520107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:56273 (local address 127.0.0.1:40247)
0927 05:11:27.520322 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:27.520331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:27.520813 (+ 482us) server_negotiation.cc:407] Connection header received
0927 05:11:27.521053 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:27.521057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:27.521105 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:27.521187 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:27.522140 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:27.522144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:27.522147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:27.522162 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:27.522199 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:27.522213 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:27.522310 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:27.522443 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 05:11:27.522493 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:11:27.529613 21769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2544211de6b14b8c83e543473d3e4695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260927 05:11:27.530027 21769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:27.530104 21769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2544211de6b14b8c83e543473d3e4695, State: Initialized, Role: FOLLOWER
I20260927 05:11:27.530650 21769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2544211de6b14b8c83e543473d3e4695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260927 05:11:27.530886 21769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:27.530992 21769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:27.531113 21769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:27.533519 21769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2544211de6b14b8c83e543473d3e4695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260927 05:11:27.534011 21769 leader_election.cc:302] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2544211de6b14b8c83e543473d3e4695; no voters:
I20260927 05:11:27.534631 21769 leader_election.cc:288] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:27.534839 21779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:27.535848 21779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [term 1 LEADER]: Becoming Leader. State: Replica: 2544211de6b14b8c83e543473d3e4695, State: Running, Role: LEADER
I20260927 05:11:27.536563 21779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2544211de6b14b8c83e543473d3e4695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260927 05:11:27.536929 21769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:27.540176 21780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2544211de6b14b8c83e543473d3e4695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2544211de6b14b8c83e543473d3e4695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } } }
I20260927 05:11:27.540731 21780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:27.541895 21781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2544211de6b14b8c83e543473d3e4695. Latest consensus state: current_term: 1 leader_uuid: "2544211de6b14b8c83e543473d3e4695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2544211de6b14b8c83e543473d3e4695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } } }
I20260927 05:11:27.542363 21781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:27.551983 21786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:27.558872 21786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:27.569763 21786 catalog_manager.cc:1347] Generated new cluster ID: efe6b2256d654bb5b1621cbf6bd4b98d
I20260927 05:11:27.569909 21786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:27.583591 21786 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:27.584169 21786 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:27.592075 21786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2544211de6b14b8c83e543473d3e4695: Generated new TSK 0
I20260927 05:11:27.593079 21786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:27.605020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21702
W20260927 05:11:27.606851 21764 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:45037
W20260927 05:11:27.607815 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21702: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:27.608017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 21702
2026-09-27T05:11:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:27 dist-test-slave-hgs0 krb5kdc[21801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:27 dist-test-slave-hgs0 krb5kdc[21801](info): set up 2 sockets
Sep 27 05:11:27 dist-test-slave-hgs0 krb5kdc[21801](info): commencing operation
krb5kdc: starting...
W20260927 05:11:28.728579 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:28 dist-test-slave-hgs0 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:29.024173 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:29.391518 21814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:29.391983 21814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:29.392174 21814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:29.399857 21814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:29.400085 21814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:29.400197 21814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:29.400309 21814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:29.400403 21814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:29.411224 21814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:29.412837 21814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:29.414392 21814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:29.428370 21819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:29.428296 21822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:29.428402 21820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:29.429776 21814 server_base.cc:1034] running on GCE node
I20260927 05:11:29.430738 21814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:29.432405 21814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:29.433656 21814 hybrid_clock.cc:648] HybridClock initialized: now 1790485889433593 us; error 58 us; skew 500 ppm
I20260927 05:11:29.434067 21814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:29 dist-test-slave-hgs0 krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:29.437531 21814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:29.439241 21814 webserver.cc:460] Webserver started at http://127.0.0.1:38237/ using document root <none> and password file <none>
I20260927 05:11:29.440058 21814 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:29.440210 21814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:29.440531 21814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:29.443703 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "095b31a1426b4d2ea8a823eac264903e"
format_stamp: "Formatted at 2026-09-27 05:11:29 on dist-test-slave-hgs0"
server_key: "d70239d1be825d66c86f492faac477c0"
server_key_iv: "4b6b2295e94ab8f0020161c3da7c3e9b"
server_key_version: "encryptionkey@0"
I20260927 05:11:29.444603 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "095b31a1426b4d2ea8a823eac264903e"
format_stamp: "Formatted at 2026-09-27 05:11:29 on dist-test-slave-hgs0"
server_key: "d70239d1be825d66c86f492faac477c0"
server_key_iv: "4b6b2295e94ab8f0020161c3da7c3e9b"
server_key_version: "encryptionkey@0"
I20260927 05:11:29.450011 21814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 05:11:29.453823 21829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:29.455327 21814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:11:29.455538 21814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "095b31a1426b4d2ea8a823eac264903e"
format_stamp: "Formatted at 2026-09-27 05:11:29 on dist-test-slave-hgs0"
server_key: "d70239d1be825d66c86f492faac477c0"
server_key_iv: "4b6b2295e94ab8f0020161c3da7c3e9b"
server_key_version: "encryptionkey@0"
I20260927 05:11:29.455760 21814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:29.470582 21814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:29.475310 21814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:29.475631 21814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:29.496955 21814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42877
I20260927 05:11:29.496997 21880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42877 every 8 connection(s)
I20260927 05:11:29.498518 21814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:29.502646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 21814
I20260927 05:11:29.502851 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:29.503151 21881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:29.503517 388 external_mini_cluster.cc:1428] Setting key fd2813fb94a8774ce245630580ee5dea
I20260927 05:11:29.518402 21881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e: Bootstrap starting.
Sep 27 05:11:29 dist-test-slave-hgs0 krb5kdc[21801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 05:11:29.522414 21881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:29.523674 21881 log.cc:824] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:29.527060 21881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e: No bootstrap required, opened a new log
I20260927 05:11:29.529580 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:29.505631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:57091 (local address 127.0.0.1:42877)
0927 05:11:29.506630 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:29.506645 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:29.506697 (+ 52us) server_negotiation.cc:407] Connection header received
0927 05:11:29.509107 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:29.509139 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:29.509720 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:29.510261 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:29.511988 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:29.513486 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:29.514912 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:29.515756 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:29.516026 (+ 270us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:29.520836 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:29.520845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:29.520864 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:29.520910 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:29.524441 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:29.525442 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:29.525451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:29.525458 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:29.525642 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:29.526200 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:29.526206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:29.526209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:29.526835 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:29.527218 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:29.527858 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 05:11:29.528267 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":402,"threads_started":1}
I20260927 05:11:29.534963 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:29.537809 21881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095b31a1426b4d2ea8a823eac264903e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260927 05:11:29.538472 21881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:29.538614 21881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095b31a1426b4d2ea8a823eac264903e, State: Initialized, Role: FOLLOWER
I20260927 05:11:29.539353 21881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095b31a1426b4d2ea8a823eac264903e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260927 05:11:29.539650 21881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:29.539835 21881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:29.540094 21881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:29.543985 21881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095b31a1426b4d2ea8a823eac264903e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260927 05:11:29.544661 21881 leader_election.cc:302] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 095b31a1426b4d2ea8a823eac264903e; no voters:
I20260927 05:11:29.545516 21881 leader_election.cc:288] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:29.545673 21891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:29.547272 21891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [term 1 LEADER]: Becoming Leader. State: Replica: 095b31a1426b4d2ea8a823eac264903e, State: Running, Role: LEADER
I20260927 05:11:29.548138 21881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:29.548065 21891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095b31a1426b4d2ea8a823eac264903e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } }
I20260927 05:11:29.551532 21892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "095b31a1426b4d2ea8a823eac264903e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095b31a1426b4d2ea8a823eac264903e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } } }
I20260927 05:11:29.552028 21892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:29.554277 21893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 095b31a1426b4d2ea8a823eac264903e. Latest consensus state: current_term: 1 leader_uuid: "095b31a1426b4d2ea8a823eac264903e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095b31a1426b4d2ea8a823eac264903e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42877 } } }
I20260927 05:11:29.554742 21893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:29.555902 21897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:29.562047 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:29.540149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:57261 (local address 127.0.0.1:42877)
0927 05:11:29.540387 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:29.540395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:29.540998 (+ 603us) server_negotiation.cc:407] Connection header received
0927 05:11:29.541192 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:29.541197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:29.541284 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:29.541406 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:29.543518 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:29.544588 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:29.546683 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:29.547447 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:29.547582 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:29.551298 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:29.551311 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:29.551318 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:29.551376 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:29.556200 (+ 4824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:29.557587 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:29.557594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:29.557597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:29.557773 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:29.558509 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:29.558522 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:29.558528 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:29.559191 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:29.559635 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:29.560971 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0927 05:11:29.561304 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 05:11:29.563786 21897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:29.578780 21897 catalog_manager.cc:1347] Generated new cluster ID: 9d9f1e60bde44028827412a60fd83283
I20260927 05:11:29.579041 21897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:29.599885 21897 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:29.602293 21897 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:29.617991 21897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 095b31a1426b4d2ea8a823eac264903e: Generated new TSK 0
I20260927 05:11:29.619240 21897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:29.632309 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21814
I20260927 05:11:29.633985 21876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:29.732446 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 21814
W20260927 05:11:29.750356 21799 connection.cc:535] client connection to 127.0.0.1:42877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42877): Connection reset by peer (error 104)
2026-09-27T05:11:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2124 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:29 dist-test-slave-hgs0 krb5kdc[21913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:29 dist-test-slave-hgs0 krb5kdc[21913](info): set up 2 sockets
Sep 27 05:11:29 dist-test-slave-hgs0 krb5kdc[21913](info): commencing operation
krb5kdc: starting...
W20260927 05:11:30.908521 388 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:31 dist-test-slave-hgs0 krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:31.207111 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:31.568125 21926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:31.568477 21926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:31.568657 21926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:31.576696 21926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:31.576846 21926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:31.576957 21926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:31.577054 21926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:31.577127 21926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:31.588078 21926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:31.589669 21926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:31.591194 21926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:31.605391 21931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:31.605448 21934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:31.605383 21932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:31.606437 21926 server_base.cc:1034] running on GCE node
I20260927 05:11:31.607517 21926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:31.609304 21926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:31.610589 21926 hybrid_clock.cc:648] HybridClock initialized: now 1790485891610537 us; error 55 us; skew 500 ppm
I20260927 05:11:31.611011 21926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:31 dist-test-slave-hgs0 krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:31.623317 21926 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:31.625273 21926 webserver.cc:460] Webserver started at http://127.0.0.1:41159/ using document root <none> and password file <none>
I20260927 05:11:31.626109 21926 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:31.626279 21926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:31.626639 21926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:31.630125 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3"
format_stamp: "Formatted at 2026-09-27 05:11:31 on dist-test-slave-hgs0"
server_key: "673a2d526f2ef22593ed2ab6335db1fe"
server_key_iv: "051d4b3ca44d68602a2f80065965ed0b"
server_key_version: "encryptionkey@0"
I20260927 05:11:31.631161 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3"
format_stamp: "Formatted at 2026-09-27 05:11:31 on dist-test-slave-hgs0"
server_key: "673a2d526f2ef22593ed2ab6335db1fe"
server_key_iv: "051d4b3ca44d68602a2f80065965ed0b"
server_key_version: "encryptionkey@0"
I20260927 05:11:31.637840 21926 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 05:11:31.642011 21941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:31.643618 21926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:11:31.643865 21926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3"
format_stamp: "Formatted at 2026-09-27 05:11:31 on dist-test-slave-hgs0"
server_key: "673a2d526f2ef22593ed2ab6335db1fe"
server_key_iv: "051d4b3ca44d68602a2f80065965ed0b"
server_key_version: "encryptionkey@0"
I20260927 05:11:31.644158 21926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:31.663228 21926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:31.669212 21926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:31.669533 21926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:31.689026 21926 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46729
I20260927 05:11:31.689047 21992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46729 every 8 connection(s)
I20260927 05:11:31.690613 21926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:31.694890 21993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:31.695055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 21926
I20260927 05:11:31.695286 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:31.695750 388 external_mini_cluster.cc:1428] Setting key 4d1007784504d80fb9c7009c19779bd4
I20260927 05:11:31.711355 21993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3: Bootstrap starting.
I20260927 05:11:31.711573 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:31.697786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:57055 (local address 127.0.0.1:46729)
0927 05:11:31.698672 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:31.698688 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:31.698750 (+ 62us) server_negotiation.cc:407] Connection header received
0927 05:11:31.700543 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:31.700582 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:31.701192 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:31.701743 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:31.702313 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:31.704419 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:31.705834 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:31.706954 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:31.707164 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:31.708667 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:31.708678 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:31.708702 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:31.709458 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:31.709555 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:31.709591 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:31.709756 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:31.710018 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 05:11:31.710214 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":371,"threads_started":1}
I20260927 05:11:31.714823 21993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:31.715646 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:31.716096 21993 log.cc:824] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:31.719527 21993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3: No bootstrap required, opened a new log
I20260927 05:11:31.728186 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:31.720874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:34417 (local address 127.0.0.1:46729)
0927 05:11:31.721057 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:31.721063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:31.721853 (+ 790us) server_negotiation.cc:407] Connection header received
0927 05:11:31.722033 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:31.722038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:31.722105 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:31.722187 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:31.722807 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:31.723590 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:31.725522 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:31.725960 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:31.726056 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:31.727360 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:31.727365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:31.727369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:31.727388 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:31.727426 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:31.727441 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:31.727528 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:31.727743 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 05:11:31.727791 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:11:31.730109 21993 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } }
I20260927 05:11:31.730489 21993 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:31.730640 21993 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8f648f51fc4bdd90bc894e7a36f9a3, State: Initialized, Role: FOLLOWER
I20260927 05:11:31.731236 21993 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } }
I20260927 05:11:31.731477 21993 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:31.731630 21993 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:31.731803 21993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:31.734764 21993 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } }
I20260927 05:11:31.735303 21993 leader_election.cc:302] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd8f648f51fc4bdd90bc894e7a36f9a3; no voters:
I20260927 05:11:31.736027 21993 leader_election.cc:288] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:31.736145 22003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:31.736902 22003 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [term 1 LEADER]: Becoming Leader. State: Replica: cd8f648f51fc4bdd90bc894e7a36f9a3, State: Running, Role: LEADER
I20260927 05:11:31.737494 22003 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } }
I20260927 05:11:31.738305 21993 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:31.740897 22005 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd8f648f51fc4bdd90bc894e7a36f9a3. Latest consensus state: current_term: 1 leader_uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } } }
I20260927 05:11:31.740937 22004 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f648f51fc4bdd90bc894e7a36f9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46729 } } }
I20260927 05:11:31.741313 22005 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:31.741389 22004 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:31.742277 22009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:31.749264 22009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:31.761592 22009 catalog_manager.cc:1347] Generated new cluster ID: 1b4495b69a664543b3d776ff43613630
I20260927 05:11:31.761737 22009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:31.783205 22009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:31.784813 22009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:31.793080 22009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd8f648f51fc4bdd90bc894e7a36f9a3: Generated new TSK 0
I20260927 05:11:31.794242 22009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:31.810397 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21926
W20260927 05:11:31.812595 21988 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:57055
W20260927 05:11:31.813516 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 21926: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:31.813675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 21926
W20260927 05:11:31.832749 21911 connection.cc:535] client connection to 127.0.0.1:46729 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46729): Connection reset by peer (error 104)
2026-09-27T05:11:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2082 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:31 dist-test-slave-hgs0 krb5kdc[22024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:31 dist-test-slave-hgs0 krb5kdc[22024](info): set up 2 sockets
Sep 27 05:11:31 dist-test-slave-hgs0 krb5kdc[22024](info): commencing operation
krb5kdc: starting...
W20260927 05:11:32.940588 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:33 dist-test-slave-hgs0 krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 05:11:33.212132 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:33.565258 22037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:33.565659 22037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:33.565846 22037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:33.573410 22037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:33.573604 22037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:33.573729 22037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:33.573841 22037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:33.573928 22037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:33.584277 22037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:33.585804 22037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:33.587301 22037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:33.601289 22042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:33.601249 22043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:33.601229 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:33.602623 22037 server_base.cc:1034] running on GCE node
I20260927 05:11:33.603580 22037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:33.605454 22037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:33.606758 22037 hybrid_clock.cc:648] HybridClock initialized: now 1790485893606691 us; error 59 us; skew 500 ppm
I20260927 05:11:33.607249 22037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:33 dist-test-slave-hgs0 krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:33.611064 22037 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 05:11:33.613095 22037 webserver.cc:460] Webserver started at http://127.0.0.1:33365/ using document root <none> and password file <none>
I20260927 05:11:33.613942 22037 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:33.614069 22037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:33.614377 22037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:33.617944 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "30188ec63bb344b1bdcce61fd3005462"
format_stamp: "Formatted at 2026-09-27 05:11:33 on dist-test-slave-hgs0"
server_key: "e28f8dde8a1ac7dcc99ee6f961a2aa0f"
server_key_iv: "0795676a06928b48cfedae4cfeca9646"
server_key_version: "encryptionkey@0"
I20260927 05:11:33.618770 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "30188ec63bb344b1bdcce61fd3005462"
format_stamp: "Formatted at 2026-09-27 05:11:33 on dist-test-slave-hgs0"
server_key: "e28f8dde8a1ac7dcc99ee6f961a2aa0f"
server_key_iv: "0795676a06928b48cfedae4cfeca9646"
server_key_version: "encryptionkey@0"
I20260927 05:11:33.624856 22037 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 05:11:33.628868 22052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:33.630575 22037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:11:33.630750 22037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "30188ec63bb344b1bdcce61fd3005462"
format_stamp: "Formatted at 2026-09-27 05:11:33 on dist-test-slave-hgs0"
server_key: "e28f8dde8a1ac7dcc99ee6f961a2aa0f"
server_key_iv: "0795676a06928b48cfedae4cfeca9646"
server_key_version: "encryptionkey@0"
I20260927 05:11:33.631054 22037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:33.637431 22037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:33.641966 22037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:33.642299 22037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:33.662360 22037 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39529
I20260927 05:11:33.662400 22103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39529 every 8 connection(s)
I20260927 05:11:33.664011 22037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:33.668114 22104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:33.668686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 22037
I20260927 05:11:33.668907 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:33.669349 388 external_mini_cluster.cc:1428] Setting key c8a5a7f4a030edf6e3b4ccd34b888025
I20260927 05:11:33.679315 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:33.671346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:39491 (local address 127.0.0.1:39529)
0927 05:11:33.672231 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:33.672249 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:33.672302 (+ 53us) server_negotiation.cc:407] Connection header received
0927 05:11:33.675038 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:33.675073 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:33.675613 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:33.676154 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:33.677191 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:33.677200 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:33.677218 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:33.677237 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:33.677309 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:33.677343 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:33.677571 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:33.677703 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 05:11:33.677951 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":331,"threads_started":1}
I20260927 05:11:33.683660 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:33.683777 22104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462: Bootstrap starting.
I20260927 05:11:33.687068 22104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:33.688325 22104 log.cc:824] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:33.690912 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:33.688228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.98:51733 (local address 127.0.0.1:39529)
0927 05:11:33.688423 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:33.688431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:33.688725 (+ 294us) server_negotiation.cc:407] Connection header received
0927 05:11:33.688880 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:33.688889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:33.688982 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:33.689122 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:33.690236 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:33.690242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:33.690246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:33.690269 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:33.690313 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:33.690333 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:33.690454 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:33.690546 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 05:11:33.690610 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 05:11:33.692096 22104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462: No bootstrap required, opened a new log
I20260927 05:11:33.700471 22104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30188ec63bb344b1bdcce61fd3005462" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39529 } }
I20260927 05:11:33.700999 22104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:33.701143 22104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30188ec63bb344b1bdcce61fd3005462, State: Initialized, Role: FOLLOWER
I20260927 05:11:33.701823 22104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30188ec63bb344b1bdcce61fd3005462" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39529 } }
I20260927 05:11:33.702112 22104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:33.702337 22104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:33.702539 22104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:33.705528 22104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30188ec63bb344b1bdcce61fd3005462" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39529 } }
I20260927 05:11:33.706162 22104 leader_election.cc:302] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30188ec63bb344b1bdcce61fd3005462; no voters:
I20260927 05:11:33.706841 22104 leader_election.cc:288] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:33.707054 22114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:33.707858 22114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [term 1 LEADER]: Becoming Leader. State: Replica: 30188ec63bb344b1bdcce61fd3005462, State: Running, Role: LEADER
I20260927 05:11:33.708537 22114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30188ec63bb344b1bdcce61fd3005462" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39529 } }
I20260927 05:11:33.709259 22104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:33.712191 22116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30188ec63bb344b1bdcce61fd3005462. Latest consensus state: current_term: 1 leader_uuid: "30188ec63bb344b1bdcce61fd3005462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30188ec63bb344b1bdcce61fd3005462" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39529 } } }
I20260927 05:11:33.712288 22115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30188ec63bb344b1bdcce61fd3005462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30188ec63bb344b1bdcce61fd3005462" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39529 } } }
I20260927 05:11:33.712594 22116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:33.712906 22115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:33.715229 22121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:33.720901 22121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:33.730755 22121 catalog_manager.cc:1347] Generated new cluster ID: 51825ed0a90e4e59b6b09b62e810d17c
I20260927 05:11:33.730888 22121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:33.770907 22121 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:33.771539 22121 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:33.782231 22121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30188ec63bb344b1bdcce61fd3005462: Generated new TSK 0
I20260927 05:11:33.783156 22121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:33.800575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22037
W20260927 05:11:33.801923 22099 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.98:39491
W20260927 05:11:33.802768 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22037: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:33.802924 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 22037
2026-09-27T05:11:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:33 dist-test-slave-hgs0 krb5kdc[22135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:33 dist-test-slave-hgs0 krb5kdc[22135](info): set up 2 sockets
Sep 27 05:11:33 dist-test-slave-hgs0 krb5kdc[22135](info): commencing operation
krb5kdc: starting...
W20260927 05:11:34.920413 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:35 dist-test-slave-hgs0 krb5kdc[22135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:11:35.187553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44527
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:35.526952 22148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:35.527323 22148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:35.527482 22148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:35.534957 22148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:35.535110 22148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:35.535218 22148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:35.535328 22148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:35.535414 22148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:35.545349 22148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:35.546747 22148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:35.548099 22148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:35.560420 22153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:35.560377 22156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:35.560451 22154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:35.561336 22148 server_base.cc:1034] running on GCE node
I20260927 05:11:35.562173 22148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:35.563621 22148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:35.564849 22148 hybrid_clock.cc:648] HybridClock initialized: now 1790485895564796 us; error 32 us; skew 500 ppm
I20260927 05:11:35.565248 22148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:35 dist-test-slave-hgs0 krb5kdc[22135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:35.568461 22148 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:11:35.569880 22148 webserver.cc:460] Webserver started at http://127.0.97.62:37025/ using document root <none> and password file <none>
I20260927 05:11:35.570528 22148 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:35.570624 22148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:35.570868 22148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:35.573791 22148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "c8cefaaadc8e4ca380e2649574212e10"
format_stamp: "Formatted at 2026-09-27 05:11:35 on dist-test-slave-hgs0"
server_key: "f61013750aa16a0443b00b0c967a69e8"
server_key_iv: "39619e9c6ba4a1a3d576feee223c0f21"
server_key_version: "encryptionkey@0"
I20260927 05:11:35.574497 22148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "c8cefaaadc8e4ca380e2649574212e10"
format_stamp: "Formatted at 2026-09-27 05:11:35 on dist-test-slave-hgs0"
server_key: "f61013750aa16a0443b00b0c967a69e8"
server_key_iv: "39619e9c6ba4a1a3d576feee223c0f21"
server_key_version: "encryptionkey@0"
I20260927 05:11:35.579483 22148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 05:11:35.582864 22163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:35.584272 22148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:11:35.584452 22148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "c8cefaaadc8e4ca380e2649574212e10"
format_stamp: "Formatted at 2026-09-27 05:11:35 on dist-test-slave-hgs0"
server_key: "f61013750aa16a0443b00b0c967a69e8"
server_key_iv: "39619e9c6ba4a1a3d576feee223c0f21"
server_key_version: "encryptionkey@0"
I20260927 05:11:35.584641 22148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:35.597162 22148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:35.601567 22148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:35.601898 22148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:35.621340 22148 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44527
I20260927 05:11:35.621362 22214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44527 every 8 connection(s)
I20260927 05:11:35.622833 22148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:35.626754 22215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:35.632649 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 22148
I20260927 05:11:35.632851 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:35.633270 388 external_mini_cluster.cc:1428] Setting key dc3a395f208b402e699a2126bc5043c2
I20260927 05:11:35.637768 22215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10: Bootstrap starting.
I20260927 05:11:35.641587 22215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:35.642870 22215 log.cc:824] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:35.645961 22215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10: No bootstrap required, opened a new log
Sep 27 05:11:35 dist-test-slave-hgs0 krb5kdc[22135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:35.655540 22215 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cefaaadc8e4ca380e2649574212e10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } }
I20260927 05:11:35.656045 22215 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:35.656167 22215 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8cefaaadc8e4ca380e2649574212e10, State: Initialized, Role: FOLLOWER
I20260927 05:11:35.657161 22215 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cefaaadc8e4ca380e2649574212e10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } }
I20260927 05:11:35.657445 22215 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:35.657601 22215 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:35.657809 22215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:35.658205 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:35.635245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48719 (local address 127.0.97.62:44527)
0927 05:11:35.636115 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:35.636127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:35.636166 (+ 39us) server_negotiation.cc:407] Connection header received
0927 05:11:35.637812 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:35.637834 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:35.638306 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:35.638627 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:35.640488 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:35.641871 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:35.643771 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:35.644581 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:35.644781 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:35.649479 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:35.649497 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:35.649523 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:35.649593 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:35.653116 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:35.654218 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:35.654231 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:35.654239 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:35.654459 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:35.654980 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:35.654984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:35.654986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:35.655646 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:35.655981 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:35.656609 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 05:11:35.656942 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":409,"threads_started":1}
I20260927 05:11:35.661055 22215 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cefaaadc8e4ca380e2649574212e10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } }
I20260927 05:11:35.661713 22215 leader_election.cc:302] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8cefaaadc8e4ca380e2649574212e10; no voters:
I20260927 05:11:35.662458 22215 leader_election.cc:288] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:35.662619 22220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:35.663225 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:35.663515 22220 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [term 1 LEADER]: Becoming Leader. State: Replica: c8cefaaadc8e4ca380e2649574212e10, State: Running, Role: LEADER
I20260927 05:11:35.664163 22220 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cefaaadc8e4ca380e2649574212e10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } }
I20260927 05:11:35.666592 22220 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8cefaaadc8e4ca380e2649574212e10. Latest consensus state: current_term: 1 leader_uuid: "c8cefaaadc8e4ca380e2649574212e10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cefaaadc8e4ca380e2649574212e10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } } }
I20260927 05:11:35.669005 22220 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:35.668422 22222 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8cefaaadc8e4ca380e2649574212e10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8cefaaadc8e4ca380e2649574212e10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44527 } } }
I20260927 05:11:35.669418 22222 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:35.671239 22215 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:35.674870 22226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:35.682796 22226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:35.698932 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:35.675389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40597 (local address 127.0.97.62:44527)
0927 05:11:35.675561 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:35.675567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:35.676605 (+ 1038us) server_negotiation.cc:407] Connection header received
0927 05:11:35.676820 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:35.676832 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:35.676943 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:35.677069 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:35.678631 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:35.679906 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:35.682494 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:35.683463 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:35.683598 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:35.688484 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:35.688495 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:35.688502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:35.688565 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:35.693837 (+ 5272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:35.695419 (+ 1582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:35.695429 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:35.695436 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:35.695645 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:35.696543 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:35.696548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:35.696551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:35.697068 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:35.697433 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:35.698020 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 05:11:35.698334 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:11:35.699376 22226 catalog_manager.cc:1347] Generated new cluster ID: ea54030eefa14829b55d712d1a35be9e
I20260927 05:11:35.699501 22226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:35.712898 22226 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:35.714962 22226 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:35.722810 22226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8cefaaadc8e4ca380e2649574212e10: Generated new TSK 0
I20260927 05:11:35.723691 22226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:35.731688 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22148
I20260927 05:11:35.732972 22210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:35.822860 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 22148
W20260927 05:11:35.837441 22133 connection.cc:535] client connection to 127.0.97.62:44527 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44527): Connection reset by peer (error 104)
2026-09-27T05:11:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:35 dist-test-slave-hgs0 krb5kdc[22247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:35 dist-test-slave-hgs0 krb5kdc[22247](info): set up 2 sockets
Sep 27 05:11:35 dist-test-slave-hgs0 krb5kdc[22247](info): commencing operation
krb5kdc: starting...
W20260927 05:11:36.920328 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:37 dist-test-slave-hgs0 krb5kdc[22247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:11:37.189551 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39331
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:37.533972 22260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:37.534318 22260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:37.534493 22260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:37.541980 22260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:37.542160 22260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:37.542249 22260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:37.542358 22260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:37.542445 22260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:37.552639 22260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:37.554145 22260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:37.555634 22260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:37.567498 22265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:37.567518 22266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:37.567517 22268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:37.568332 22260 server_base.cc:1034] running on GCE node
I20260927 05:11:37.569124 22260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:37.570516 22260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:37.571775 22260 hybrid_clock.cc:648] HybridClock initialized: now 1790485897571715 us; error 72 us; skew 500 ppm
I20260927 05:11:37.572180 22260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:37 dist-test-slave-hgs0 krb5kdc[22247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:37.579056 22260 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:11:37.580753 22260 webserver.cc:460] Webserver started at http://127.0.97.62:36685/ using document root <none> and password file <none>
I20260927 05:11:37.581523 22260 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:37.581688 22260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:37.582011 22260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:37.585182 22260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "c6a82bcb64154044b0463ce0979ec813"
format_stamp: "Formatted at 2026-09-27 05:11:37 on dist-test-slave-hgs0"
server_key: "f08e6c807b41f74621b8f5dcc1d18bf5"
server_key_iv: "df95d1bf1195b1aa13da0d334525e77a"
server_key_version: "encryptionkey@0"
I20260927 05:11:37.585997 22260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "c6a82bcb64154044b0463ce0979ec813"
format_stamp: "Formatted at 2026-09-27 05:11:37 on dist-test-slave-hgs0"
server_key: "f08e6c807b41f74621b8f5dcc1d18bf5"
server_key_iv: "df95d1bf1195b1aa13da0d334525e77a"
server_key_version: "encryptionkey@0"
I20260927 05:11:37.591653 22260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 05:11:37.595633 22275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:37.597299 22260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:11:37.597527 22260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "c6a82bcb64154044b0463ce0979ec813"
format_stamp: "Formatted at 2026-09-27 05:11:37 on dist-test-slave-hgs0"
server_key: "f08e6c807b41f74621b8f5dcc1d18bf5"
server_key_iv: "df95d1bf1195b1aa13da0d334525e77a"
server_key_version: "encryptionkey@0"
I20260927 05:11:37.597767 22260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:37.620882 22260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:37.625855 22260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:37.626178 22260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:37.645618 22260 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39331
I20260927 05:11:37.645720 22326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39331 every 8 connection(s)
I20260927 05:11:37.647101 22260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:37.650972 22327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:37.656209 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 22260
I20260927 05:11:37.656385 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:37.656805 388 external_mini_cluster.cc:1428] Setting key daa446aa516bdd6c0b92dff6ebfba1df
I20260927 05:11:37.662619 22327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813: Bootstrap starting.
I20260927 05:11:37.666483 22327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:37.667790 22327 log.cc:824] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:37.671141 22327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813: No bootstrap required, opened a new log
I20260927 05:11:37.671505 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:37.658645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51107 (local address 127.0.97.62:39331)
0927 05:11:37.659539 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:37.659553 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:37.659598 (+ 45us) server_negotiation.cc:407] Connection header received
0927 05:11:37.661437 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:37.661464 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:37.661914 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:37.662270 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:37.662922 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:37.664334 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:37.666302 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:37.667105 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:37.667251 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:37.668789 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:37.668803 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:37.668832 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:37.669587 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:37.669664 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:37.669689 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:37.669828 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:37.670200 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 05:11:37.670419 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":428,"threads_started":1}
I20260927 05:11:37.675616 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:37.679852 22327 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a82bcb64154044b0463ce0979ec813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39331 } }
I20260927 05:11:37.680711 22327 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:37.680847 22327 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a82bcb64154044b0463ce0979ec813, State: Initialized, Role: FOLLOWER
I20260927 05:11:37.681645 22327 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a82bcb64154044b0463ce0979ec813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39331 } }
I20260927 05:11:37.681933 22327 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:37.682147 22327 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:37.682377 22327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:37.685166 22327 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a82bcb64154044b0463ce0979ec813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39331 } }
I20260927 05:11:37.685837 22327 leader_election.cc:302] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6a82bcb64154044b0463ce0979ec813; no voters:
I20260927 05:11:37.686569 22327 leader_election.cc:288] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:37.686759 22337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:37.688036 22337 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [term 1 LEADER]: Becoming Leader. State: Replica: c6a82bcb64154044b0463ce0979ec813, State: Running, Role: LEADER
I20260927 05:11:37.688619 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:37.680403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57259 (local address 127.0.97.62:39331)
0927 05:11:37.680644 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:37.680653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:37.681132 (+ 479us) server_negotiation.cc:407] Connection header received
0927 05:11:37.681409 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:37.681414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:37.681488 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:37.681612 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:37.682357 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:37.683329 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:37.684963 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:37.685552 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:37.685719 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:37.687289 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:37.687300 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:37.687312 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:37.687353 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:37.687487 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:37.687533 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:37.687682 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:37.688172 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 05:11:37.688248 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260927 05:11:37.688819 22337 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a82bcb64154044b0463ce0979ec813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39331 } }
I20260927 05:11:37.689293 22327 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:37.693084 22339 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6a82bcb64154044b0463ce0979ec813. Latest consensus state: current_term: 1 leader_uuid: "c6a82bcb64154044b0463ce0979ec813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a82bcb64154044b0463ce0979ec813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39331 } } }
I20260927 05:11:37.693589 22339 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:37.694142 22338 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6a82bcb64154044b0463ce0979ec813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a82bcb64154044b0463ce0979ec813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39331 } } }
I20260927 05:11:37.694437 22338 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:37.696871 22346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:37.703001 22346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:37.714121 22346 catalog_manager.cc:1347] Generated new cluster ID: 7209c0d704e24d1eb14cba780dcbb209
I20260927 05:11:37.714278 22346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:37.733180 22346 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:37.734532 22346 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:37.740099 22346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6a82bcb64154044b0463ce0979ec813: Generated new TSK 0
I20260927 05:11:37.741122 22346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:37.756101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22260
W20260927 05:11:37.757225 22322 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51107
W20260927 05:11:37.758024 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22260: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:37.758186 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 22260
W20260927 05:11:37.772755 22245 connection.cc:535] client connection to 127.0.97.62:39331 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:39331): Connection reset by peer (error 104)
2026-09-27T05:11:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1935 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:37 dist-test-slave-hgs0 krb5kdc[22358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:37 dist-test-slave-hgs0 krb5kdc[22358](info): set up 2 sockets
Sep 27 05:11:37 dist-test-slave-hgs0 krb5kdc[22358](info): commencing operation
krb5kdc: starting...
W20260927 05:11:38.848426 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:39 dist-test-slave-hgs0 krb5kdc[22358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:11:39.120062 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40655
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:39.455971 22371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:39.456264 22371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:39.456426 22371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:39.463907 22371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:39.464054 22371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:39.464131 22371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:39.464234 22371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:39.464320 22371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:39.474323 22371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:39.475666 22371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:39.476982 22371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:39.488778 22376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:39.488821 22379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:39.488860 22377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:39.490067 22371 server_base.cc:1034] running on GCE node
I20260927 05:11:39.490835 22371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:39.492254 22371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:39.493535 22371 hybrid_clock.cc:648] HybridClock initialized: now 1790485899493457 us; error 51 us; skew 500 ppm
I20260927 05:11:39.493942 22371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:39 dist-test-slave-hgs0 krb5kdc[22358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:39.497200 22371 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:11:39.498605 22371 webserver.cc:460] Webserver started at http://127.0.97.62:35997/ using document root <none> and password file <none>
I20260927 05:11:39.499306 22371 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:39.499452 22371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:39.499758 22371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:39.502818 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677"
format_stamp: "Formatted at 2026-09-27 05:11:39 on dist-test-slave-hgs0"
server_key: "43a671d7a95ee48def21960e5bab0272"
server_key_iv: "34f2a73b64774b0384a0feaa66173d92"
server_key_version: "encryptionkey@0"
I20260927 05:11:39.503594 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677"
format_stamp: "Formatted at 2026-09-27 05:11:39 on dist-test-slave-hgs0"
server_key: "43a671d7a95ee48def21960e5bab0272"
server_key_iv: "34f2a73b64774b0384a0feaa66173d92"
server_key_version: "encryptionkey@0"
I20260927 05:11:39.509078 22371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 05:11:39.512720 22386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:39.514181 22371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 05:11:39.514400 22371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677"
format_stamp: "Formatted at 2026-09-27 05:11:39 on dist-test-slave-hgs0"
server_key: "43a671d7a95ee48def21960e5bab0272"
server_key_iv: "34f2a73b64774b0384a0feaa66173d92"
server_key_version: "encryptionkey@0"
I20260927 05:11:39.514636 22371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:39.520944 22371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:39.526108 22371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:39.526438 22371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:39.545183 22371 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40655
I20260927 05:11:39.545234 22437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40655 every 8 connection(s)
I20260927 05:11:39.546638 22371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:39.550395 22438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:39.556100 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 22371
I20260927 05:11:39.556282 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:39.556838 388 external_mini_cluster.cc:1428] Setting key 698c5bfd8374cea7c50bbc2471812858
I20260927 05:11:39.561854 22438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677: Bootstrap starting.
I20260927 05:11:39.564793 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:39.558674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55083 (local address 127.0.97.62:40655)
0927 05:11:39.559606 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:39.559638 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:39.559702 (+ 64us) server_negotiation.cc:407] Connection header received
0927 05:11:39.561402 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:39.561425 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:39.561769 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:39.562143 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:39.563155 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:39.563162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:39.563175 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:39.563190 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:39.563249 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:39.563269 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:39.563431 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:39.563553 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 05:11:39.563779 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":424,"threads_started":1}
I20260927 05:11:39.565850 22438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:39.566951 22438 log.cc:824] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:39.568650 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:39.570415 22438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677: No bootstrap required, opened a new log
I20260927 05:11:39.576534 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:39.574038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34367 (local address 127.0.97.62:40655)
0927 05:11:39.574263 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:39.574271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:39.574586 (+ 315us) server_negotiation.cc:407] Connection header received
0927 05:11:39.574742 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:39.574745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:39.574796 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:39.574880 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:39.575798 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:39.575803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:39.575806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:39.575820 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:39.575862 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:39.575876 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:39.576053 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:39.576143 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 05:11:39.576194 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:11:39.580124 22438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40655 } }
I20260927 05:11:39.580523 22438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:39.580647 22438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4167867b5a0d4ddcbdf1a69e2ec6e677, State: Initialized, Role: FOLLOWER
I20260927 05:11:39.581426 22438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40655 } }
I20260927 05:11:39.581693 22438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:39.581843 22438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:39.582026 22438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:39.584286 22438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40655 } }
I20260927 05:11:39.584803 22438 leader_election.cc:302] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4167867b5a0d4ddcbdf1a69e2ec6e677; no voters:
I20260927 05:11:39.585422 22438 leader_election.cc:288] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:39.585578 22448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:39.586352 22448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [term 1 LEADER]: Becoming Leader. State: Replica: 4167867b5a0d4ddcbdf1a69e2ec6e677, State: Running, Role: LEADER
I20260927 05:11:39.586966 22448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40655 } }
I20260927 05:11:39.587915 22438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:39.590480 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40655 } } }
I20260927 05:11:39.590890 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:39.591651 22450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4167867b5a0d4ddcbdf1a69e2ec6e677. Latest consensus state: current_term: 1 leader_uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4167867b5a0d4ddcbdf1a69e2ec6e677" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40655 } } }
I20260927 05:11:39.592103 22450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:39.593693 22455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:39.598433 22455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:39.606967 22455 catalog_manager.cc:1347] Generated new cluster ID: d2840f4fd14145fa88b453d9927e110c
I20260927 05:11:39.607096 22455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:39.643700 22455 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:39.644251 22455 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:39.659508 22455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4167867b5a0d4ddcbdf1a69e2ec6e677: Generated new TSK 0
I20260927 05:11:39.660542 22455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:39.667145 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22371
W20260927 05:11:39.668391 22433 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55083
W20260927 05:11:39.669168 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22371: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:39.669332 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 22371
2026-09-27T05:11:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:39 dist-test-slave-hgs0 krb5kdc[22469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:39 dist-test-slave-hgs0 krb5kdc[22469](info): set up 2 sockets
Sep 27 05:11:39 dist-test-slave-hgs0 krb5kdc[22469](info): commencing operation
krb5kdc: starting...
W20260927 05:11:40.840536 388 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:41 dist-test-slave-hgs0 krb5kdc[22469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:11:41.116547 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44677
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:41.466907 22482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:41.467283 22482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:41.467458 22482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:41.474828 22482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:41.475005 22482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:41.475117 22482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:41.475229 22482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:41.475317 22482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:41.485584 22482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:41.487031 22482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:41.488464 22482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:41.500710 22487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:41.500764 22488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:41.500921 22490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:41.501719 22482 server_base.cc:1034] running on GCE node
I20260927 05:11:41.502518 22482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:41.504096 22482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:41.505329 22482 hybrid_clock.cc:648] HybridClock initialized: now 1790485901505279 us; error 44 us; skew 500 ppm
I20260927 05:11:41.505697 22482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:41 dist-test-slave-hgs0 krb5kdc[22469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:41.509003 22482 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:11:41.510586 22482 webserver.cc:460] Webserver started at http://127.0.97.62:41957/ using document root <none> and password file <none>
I20260927 05:11:41.511365 22482 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:41.511518 22482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:41.511840 22482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:41.514961 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "d02ca06258b2430a9b1142b48187b67a"
format_stamp: "Formatted at 2026-09-27 05:11:41 on dist-test-slave-hgs0"
server_key: "16459918f6d86d6acfc376c7e921ef1f"
server_key_iv: "e9e231c303abca9211d5acb4541db3e0"
server_key_version: "encryptionkey@0"
I20260927 05:11:41.515789 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "d02ca06258b2430a9b1142b48187b67a"
format_stamp: "Formatted at 2026-09-27 05:11:41 on dist-test-slave-hgs0"
server_key: "16459918f6d86d6acfc376c7e921ef1f"
server_key_iv: "e9e231c303abca9211d5acb4541db3e0"
server_key_version: "encryptionkey@0"
I20260927 05:11:41.521919 22482 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 05:11:41.526015 22497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:41.527622 22482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 05:11:41.527853 22482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "d02ca06258b2430a9b1142b48187b67a"
format_stamp: "Formatted at 2026-09-27 05:11:41 on dist-test-slave-hgs0"
server_key: "16459918f6d86d6acfc376c7e921ef1f"
server_key_iv: "e9e231c303abca9211d5acb4541db3e0"
server_key_version: "encryptionkey@0"
I20260927 05:11:41.528218 22482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:41.551751 22482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:41.557195 22482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:41.557591 22482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:41.579370 22482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44677
I20260927 05:11:41.579394 22548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44677 every 8 connection(s)
I20260927 05:11:41.581102 22482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:41.582664 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 22482
I20260927 05:11:41.582855 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:41.583407 388 external_mini_cluster.cc:1428] Setting key 3c6fb332dcf24740e5e95cedc30bc535
I20260927 05:11:41.586284 22549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 05:11:41 dist-test-slave-hgs0 krb5kdc[22469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:41.597867 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a: Bootstrap starting.
I20260927 05:11:41.601692 22549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:41.602838 22549 log.cc:824] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:41.606381 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a: No bootstrap required, opened a new log
I20260927 05:11:41.606950 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:41.585495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45045 (local address 127.0.97.62:44677)
0927 05:11:41.586336 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:41.586386 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:41.586447 (+ 61us) server_negotiation.cc:407] Connection header received
0927 05:11:41.588256 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:41.588279 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:41.588698 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:41.589123 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:41.590597 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:41.592006 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:41.593303 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:41.594009 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:41.594247 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:41.598813 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:41.598822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:41.598838 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:41.598880 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:41.602330 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:41.603149 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:41.603159 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:41.603168 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:41.603412 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:41.604023 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:41.604027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:41.604030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:41.604674 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:41.604907 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:41.605393 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 05:11:41.605787 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":366,"threads_started":1}
I20260927 05:11:41.611280 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:41.617645 22549 raft_consensus.cc:348] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02ca06258b2430a9b1142b48187b67a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44677 } }
I20260927 05:11:41.618255 22549 raft_consensus.cc:374] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:41.618402 22549 raft_consensus.cc:729] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02ca06258b2430a9b1142b48187b67a, State: Initialized, Role: FOLLOWER
I20260927 05:11:41.619256 22549 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02ca06258b2430a9b1142b48187b67a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44677 } }
I20260927 05:11:41.619611 22549 raft_consensus.cc:388] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:41.619772 22549 raft_consensus.cc:482] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:41.620044 22549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:41.623536 22549 raft_consensus.cc:504] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02ca06258b2430a9b1142b48187b67a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44677 } }
I20260927 05:11:41.624310 22549 leader_election.cc:302] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d02ca06258b2430a9b1142b48187b67a; no voters:
I20260927 05:11:41.625188 22549 leader_election.cc:288] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:41.625314 22559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:41.626679 22559 raft_consensus.cc:686] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [term 1 LEADER]: Becoming Leader. State: Replica: d02ca06258b2430a9b1142b48187b67a, State: Running, Role: LEADER
I20260927 05:11:41.627386 22559 consensus_queue.cc:237] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02ca06258b2430a9b1142b48187b67a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44677 } }
I20260927 05:11:41.627715 22549 sys_catalog.cc:564] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:41.631906 22561 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02ca06258b2430a9b1142b48187b67a. Latest consensus state: current_term: 1 leader_uuid: "d02ca06258b2430a9b1142b48187b67a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02ca06258b2430a9b1142b48187b67a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44677 } } }
I20260927 05:11:41.632421 22561 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:41.633013 22560 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d02ca06258b2430a9b1142b48187b67a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02ca06258b2430a9b1142b48187b67a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44677 } } }
I20260927 05:11:41.633378 22560 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:41.637276 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:41.616806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37493 (local address 127.0.97.62:44677)
0927 05:11:41.617069 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:41.617080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:41.617662 (+ 582us) server_negotiation.cc:407] Connection header received
0927 05:11:41.617868 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:41.617872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:41.617942 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:41.618035 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:41.620369 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:41.621271 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:41.623350 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:41.624003 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:41.624128 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:41.627242 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:41.627250 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:41.627254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:41.627297 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:41.632094 (+ 4797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:41.633079 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:41.633085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:41.633088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:41.633311 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:41.633835 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:41.633843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:41.633850 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:41.634373 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:41.634752 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:41.636418 (+ 1666us) server_negotiation.cc:299] Negotiation successful
0927 05:11:41.636752 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 05:11:41.641230 22566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:41.645996 22566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:41.655377 22566 catalog_manager.cc:1347] Generated new cluster ID: 5053db50a8f148d59881a8803368a47f
I20260927 05:11:41.655586 22566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:41.665982 22566 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:41.667961 22566 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:41.681684 22566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d02ca06258b2430a9b1142b48187b67a: Generated new TSK 0
I20260927 05:11:41.682672 22566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:41.691725 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22482
I20260927 05:11:41.692816 22544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:41.785846 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 22482
2026-09-27T05:11:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2117 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:41 dist-test-slave-hgs0 krb5kdc[22581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:41 dist-test-slave-hgs0 krb5kdc[22581](info): set up 2 sockets
Sep 27 05:11:41 dist-test-slave-hgs0 krb5kdc[22581](info): commencing operation
krb5kdc: starting...
W20260927 05:11:42.892396 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:43 dist-test-slave-hgs0 krb5kdc[22581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:11:43.165145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34245
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:43.515148 22594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:43.515439 22594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:43.515573 22594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:43.522953 22594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:43.523077 22594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:43.523195 22594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:43.523313 22594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:43.523397 22594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:43.533998 22594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:43.535384 22594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:43.536800 22594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:43.549151 22599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:43.549114 22602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:43.549151 22600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:43.550333 22594 server_base.cc:1034] running on GCE node
I20260927 05:11:43.551208 22594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:43.552767 22594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:43.554018 22594 hybrid_clock.cc:648] HybridClock initialized: now 1790485903553974 us; error 42 us; skew 500 ppm
I20260927 05:11:43.554392 22594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:43 dist-test-slave-hgs0 krb5kdc[22581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:43.561259 22594 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:11:43.562950 22594 webserver.cc:460] Webserver started at http://127.0.97.62:40273/ using document root <none> and password file <none>
I20260927 05:11:43.563700 22594 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:43.563843 22594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:43.564208 22594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:43.567341 22594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "b6c205e744314c9296eca0a28b400935"
format_stamp: "Formatted at 2026-09-27 05:11:43 on dist-test-slave-hgs0"
server_key: "4cd0511e988d49336dec2aab2f2eb196"
server_key_iv: "e71c8f22083a80b0b3c84ddaeeb8d2a3"
server_key_version: "encryptionkey@0"
I20260927 05:11:43.568228 22594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "b6c205e744314c9296eca0a28b400935"
format_stamp: "Formatted at 2026-09-27 05:11:43 on dist-test-slave-hgs0"
server_key: "4cd0511e988d49336dec2aab2f2eb196"
server_key_iv: "e71c8f22083a80b0b3c84ddaeeb8d2a3"
server_key_version: "encryptionkey@0"
I20260927 05:11:43.573797 22594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 05:11:43.577608 22609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:43.579200 22594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 05:11:43.579417 22594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "b6c205e744314c9296eca0a28b400935"
format_stamp: "Formatted at 2026-09-27 05:11:43 on dist-test-slave-hgs0"
server_key: "4cd0511e988d49336dec2aab2f2eb196"
server_key_iv: "e71c8f22083a80b0b3c84ddaeeb8d2a3"
server_key_version: "encryptionkey@0"
I20260927 05:11:43.579650 22594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:43.599648 22594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:43.604489 22594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:43.604815 22594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:43.624491 22594 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34245
I20260927 05:11:43.624511 22660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34245 every 8 connection(s)
I20260927 05:11:43.625959 22594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:43.629946 22661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:43.632375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 22594
I20260927 05:11:43.632598 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:43.633178 388 external_mini_cluster.cc:1428] Setting key 66fa7b34b2a7631947c6008105049bbc
I20260927 05:11:43.642973 22661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935: Bootstrap starting.
I20260927 05:11:43.646494 22661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:43.647548 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:43.635398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55501 (local address 127.0.97.62:34245)
0927 05:11:43.636310 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:43.636324 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:43.636372 (+ 48us) server_negotiation.cc:407] Connection header received
0927 05:11:43.638240 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:43.638266 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:43.638725 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:43.639181 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:43.639818 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:43.641335 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:43.642726 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:43.643563 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:43.643786 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:43.645136 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:43.645143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:43.645160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:43.645726 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:43.645806 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:43.645837 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:43.645966 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:43.646205 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 05:11:43.646494 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":354,"threads_started":1}
I20260927 05:11:43.647657 22661 log.cc:824] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:43.650954 22661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935: No bootstrap required, opened a new log
I20260927 05:11:43.651341 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:43.659744 22661 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c205e744314c9296eca0a28b400935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34245 } }
I20260927 05:11:43.660295 22661 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:43.660429 22661 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c205e744314c9296eca0a28b400935, State: Initialized, Role: FOLLOWER
I20260927 05:11:43.661098 22661 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c205e744314c9296eca0a28b400935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34245 } }
I20260927 05:11:43.661396 22661 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:43.661558 22661 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:43.661737 22661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:43.662667 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:43.655699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33805 (local address 127.0.97.62:34245)
0927 05:11:43.655923 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:43.655929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:43.656365 (+ 436us) server_negotiation.cc:407] Connection header received
0927 05:11:43.656514 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:43.656519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:43.656616 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:43.656732 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:43.657203 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:43.658374 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:43.659747 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:43.660444 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:43.660594 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:43.661937 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:43.661942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:43.661945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:43.661964 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:43.662002 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:43.662022 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:43.662132 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:43.662254 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 05:11:43.662356 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:11:43.664551 22661 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c205e744314c9296eca0a28b400935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34245 } }
I20260927 05:11:43.665074 22661 leader_election.cc:302] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c205e744314c9296eca0a28b400935; no voters:
I20260927 05:11:43.665766 22661 leader_election.cc:288] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:43.665915 22671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:43.666682 22671 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [term 1 LEADER]: Becoming Leader. State: Replica: b6c205e744314c9296eca0a28b400935, State: Running, Role: LEADER
I20260927 05:11:43.667294 22671 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c205e744314c9296eca0a28b400935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34245 } }
I20260927 05:11:43.668397 22661 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:43.670178 22673 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c205e744314c9296eca0a28b400935. Latest consensus state: current_term: 1 leader_uuid: "b6c205e744314c9296eca0a28b400935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c205e744314c9296eca0a28b400935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34245 } } }
I20260927 05:11:43.670547 22673 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:43.675130 22672 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6c205e744314c9296eca0a28b400935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c205e744314c9296eca0a28b400935" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34245 } } }
I20260927 05:11:43.675607 22672 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:43.676525 22676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:43.681860 22676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:43.691566 22676 catalog_manager.cc:1347] Generated new cluster ID: 9f11ade224134c95978805355d1360f3
I20260927 05:11:43.691751 22676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:43.704691 22676 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:43.706681 22676 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:43.719240 22676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6c205e744314c9296eca0a28b400935: Generated new TSK 0
I20260927 05:11:43.720386 22676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:43.728339 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22594
W20260927 05:11:43.729571 22656 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55501
W20260927 05:11:43.730244 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22594: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:43.730388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 22594
2026-09-27T05:11:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:43 dist-test-slave-hgs0 krb5kdc[22692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:43 dist-test-slave-hgs0 krb5kdc[22692](info): set up 2 sockets
Sep 27 05:11:43 dist-test-slave-hgs0 krb5kdc[22692](info): commencing operation
krb5kdc: starting...
W20260927 05:11:44.844442 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:45 dist-test-slave-hgs0 krb5kdc[22692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:11:45.106935 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45605
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:45.436066 22705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:45.436373 22705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:45.436545 22705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:45.444176 22705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:45.444293 22705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:45.444402 22705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:45.444519 22705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:45.444597 22705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:45.454770 22705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:45.456193 22705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:45.457494 22705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:45.469380 22711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:45.469429 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:45.469327 22710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:45.470105 22705 server_base.cc:1034] running on GCE node
I20260927 05:11:45.470973 22705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:45.472519 22705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:45.473754 22705 hybrid_clock.cc:648] HybridClock initialized: now 1790485905473692 us; error 61 us; skew 500 ppm
I20260927 05:11:45.474123 22705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:45 dist-test-slave-hgs0 krb5kdc[22692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:45.477530 22705 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:11:45.479240 22705 webserver.cc:460] Webserver started at http://127.0.97.62:37609/ using document root <none> and password file <none>
I20260927 05:11:45.480067 22705 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:45.480221 22705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:45.480541 22705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:45.483629 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "adec39df94654917b5d12b5109ac6e95"
format_stamp: "Formatted at 2026-09-27 05:11:45 on dist-test-slave-hgs0"
server_key: "4116de9043487b9265449e85a3a8de62"
server_key_iv: "8fbf6edabc46f817fdb304d16eadda25"
server_key_version: "encryptionkey@0"
I20260927 05:11:45.484591 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "adec39df94654917b5d12b5109ac6e95"
format_stamp: "Formatted at 2026-09-27 05:11:45 on dist-test-slave-hgs0"
server_key: "4116de9043487b9265449e85a3a8de62"
server_key_iv: "8fbf6edabc46f817fdb304d16eadda25"
server_key_version: "encryptionkey@0"
I20260927 05:11:45.490305 22705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:11:45.494058 22720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:45.495469 22705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 05:11:45.495669 22705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "adec39df94654917b5d12b5109ac6e95"
format_stamp: "Formatted at 2026-09-27 05:11:45 on dist-test-slave-hgs0"
server_key: "4116de9043487b9265449e85a3a8de62"
server_key_iv: "8fbf6edabc46f817fdb304d16eadda25"
server_key_version: "encryptionkey@0"
I20260927 05:11:45.495888 22705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:45.501736 22705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:45.506219 22705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:45.506532 22705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:45.525245 22705 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45605
I20260927 05:11:45.525288 22771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45605 every 8 connection(s)
I20260927 05:11:45.526759 22705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:45.530575 22772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:45.531899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 22705
I20260927 05:11:45.532094 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:45.532663 388 external_mini_cluster.cc:1428] Setting key 6b3cf4ba696251b84f6eb4af8982f448
I20260927 05:11:45.540666 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:45.534498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37911 (local address 127.0.97.62:45605)
0927 05:11:45.535255 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:45.535267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:45.535309 (+ 42us) server_negotiation.cc:407] Connection header received
0927 05:11:45.537141 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:45.537167 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:45.537548 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:45.537973 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:45.538999 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:45.539005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:45.539018 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:45.539031 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:45.539089 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:45.539108 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:45.539337 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:45.539468 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 05:11:45.539647 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":325,"threads_started":1}
I20260927 05:11:45.544107 22772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95: Bootstrap starting.
I20260927 05:11:45.544718 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:45.547374 22772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:45.548625 22772 log.cc:824] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:45.551800 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:45.549115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47423 (local address 127.0.97.62:45605)
0927 05:11:45.549351 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:45.549360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:45.549628 (+ 268us) server_negotiation.cc:407] Connection header received
0927 05:11:45.549780 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:45.549790 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:45.549873 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:45.550014 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:45.550959 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:45.550968 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:45.550976 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 05:11:45.551001 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:45.551063 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 05:11:45.551091 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:45.551253 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:45.551306 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 05:11:45.551377 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 05:11:45.555022 22772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95: No bootstrap required, opened a new log
I20260927 05:11:45.562954 22772 raft_consensus.cc:348] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adec39df94654917b5d12b5109ac6e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45605 } }
I20260927 05:11:45.563354 22772 raft_consensus.cc:374] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 05:11:45.563419 22772 raft_consensus.cc:729] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adec39df94654917b5d12b5109ac6e95, State: Initialized, Role: FOLLOWER
I20260927 05:11:45.564006 22772 consensus_queue.cc:260] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adec39df94654917b5d12b5109ac6e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45605 } }
I20260927 05:11:45.564231 22772 raft_consensus.cc:388] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 05:11:45.564333 22772 raft_consensus.cc:482] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 05:11:45.564457 22772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:45.566567 22772 raft_consensus.cc:504] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adec39df94654917b5d12b5109ac6e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45605 } }
I20260927 05:11:45.567034 22772 leader_election.cc:302] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adec39df94654917b5d12b5109ac6e95; no voters:
I20260927 05:11:45.567724 22772 leader_election.cc:288] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 05:11:45.567852 22782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:45.570029 22772 sys_catalog.cc:564] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:45.570492 22782 raft_consensus.cc:686] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [term 1 LEADER]: Becoming Leader. State: Replica: adec39df94654917b5d12b5109ac6e95, State: Running, Role: LEADER
I20260927 05:11:45.571030 22782 consensus_queue.cc:237] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adec39df94654917b5d12b5109ac6e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45605 } }
I20260927 05:11:45.574391 22784 sys_catalog.cc:455] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adec39df94654917b5d12b5109ac6e95. Latest consensus state: current_term: 1 leader_uuid: "adec39df94654917b5d12b5109ac6e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adec39df94654917b5d12b5109ac6e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45605 } } }
I20260927 05:11:45.574386 22783 sys_catalog.cc:455] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adec39df94654917b5d12b5109ac6e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adec39df94654917b5d12b5109ac6e95" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45605 } } }
I20260927 05:11:45.574846 22784 sys_catalog.cc:458] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:45.574855 22783 sys_catalog.cc:458] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:45.575841 22792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:45.581933 22792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:45.591995 22792 catalog_manager.cc:1347] Generated new cluster ID: cf0567f26a8c46cd8b1208dd7bd4d67c
I20260927 05:11:45.592145 22792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:45.614061 22792 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:45.614661 22792 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:45.627909 22792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adec39df94654917b5d12b5109ac6e95: Generated new TSK 0
I20260927 05:11:45.629084 22792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:45.645768 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22705
W20260927 05:11:45.646946 22767 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37911
W20260927 05:11:45.647703 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22705: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 05:11:45.647847 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 22705
2026-09-27T05:11:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1920 ms)
[----------] 18 tests from AuthTokenIssuingTest (36255 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:45 dist-test-slave-hgs0 krb5kdc[22803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:45 dist-test-slave-hgs0 krb5kdc[22803](info): set up 2 sockets
Sep 27 05:11:45 dist-test-slave-hgs0 krb5kdc[22803](info): commencing operation
krb5kdc: starting...
W20260927 05:11:46.800632 388 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:46 dist-test-slave-hgs0 krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:11:47.080518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44343
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44343,127.0.97.61:39715,127.0.97.60:36689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:47.413720 22816 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:11:47.414207 22816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:47.414358 22816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:47.414543 22816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:47.422653 22816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:47.422811 22816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:47.422917 22816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:47.423019 22816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:47.423111 22816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:47.433279 22816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44343,127.0.97.61:39715,127.0.97.60:36689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:47.434726 22816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:47.436158 22816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:47.448204 22821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:47.448227 22822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:47.448167 22824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:47.449023 22816 server_base.cc:1034] running on GCE node
I20260927 05:11:47.449779 22816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:47.451161 22816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:47.452404 22816 hybrid_clock.cc:648] HybridClock initialized: now 1790485907452350 us; error 39 us; skew 500 ppm
I20260927 05:11:47.452759 22816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:47 dist-test-slave-hgs0 krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:47.456087 22816 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:11:47.457532 22816 webserver.cc:460] Webserver started at http://127.0.97.62:41725/ using document root <none> and password file <none>
I20260927 05:11:47.458280 22816 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:47.458418 22816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:47.458747 22816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:47.461849 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "62ed78094dd444ac8a3b09802bc6e573"
format_stamp: "Formatted at 2026-09-27 05:11:47 on dist-test-slave-hgs0"
server_key: "be9469d9033c9e53f279381144441da0"
server_key_iv: "5ca8726425b7efbcd0db3def2fa24b8f"
server_key_version: "encryptionkey@0"
I20260927 05:11:47.462671 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "62ed78094dd444ac8a3b09802bc6e573"
format_stamp: "Formatted at 2026-09-27 05:11:47 on dist-test-slave-hgs0"
server_key: "be9469d9033c9e53f279381144441da0"
server_key_iv: "5ca8726425b7efbcd0db3def2fa24b8f"
server_key_version: "encryptionkey@0"
I20260927 05:11:47.467881 22816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 05:11:47.471467 22831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:47.472909 22816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:11:47.473109 22816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "62ed78094dd444ac8a3b09802bc6e573"
format_stamp: "Formatted at 2026-09-27 05:11:47 on dist-test-slave-hgs0"
server_key: "be9469d9033c9e53f279381144441da0"
server_key_iv: "5ca8726425b7efbcd0db3def2fa24b8f"
server_key_version: "encryptionkey@0"
I20260927 05:11:47.473352 22816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:47.487567 22816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:47.491581 22816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:47.491897 22816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:47.510797 22816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44343
I20260927 05:11:47.510804 22882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44343 every 8 connection(s)
I20260927 05:11:47.512315 22816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:47.515280 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 22816
I20260927 05:11:47.515506 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:47.516292 388 external_mini_cluster.cc:1428] Setting key 94be43f32916b479d853123b6e6e378a
I20260927 05:11:47.516921 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:47.527580 22883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 27 05:11:47 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:47.560459 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.531246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.0.97.62:44343)
0927 05:11:47.533436 (+ 2190us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:47.533443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:47.533500 (+ 57us) server_negotiation.cc:407] Connection header received
0927 05:11:47.535928 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:47.536000 (+ 72us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:47.536323 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:47.536502 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:47.538385 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:47.539703 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:47.542211 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:47.542799 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:47.542958 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:47.550131 (+ 7173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:47.550147 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:47.550168 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:47.550222 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:47.554852 (+ 4630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:47.555996 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:47.556010 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:47.556018 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:47.556278 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:47.556835 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:47.556844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:47.556851 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:47.557804 (+ 953us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:47.558041 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:47.558775 (+ 734us) server_negotiation.cc:299] Negotiation successful
0927 05:11:47.559288 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":1969,"spinlock_wait_cycles":25856,"thread_start_us":1712,"threads_started":1}
I20260927 05:11:47.560573 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.531246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44343 (local address 127.0.0.1:52186)
0927 05:11:47.532295 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.532368 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:47.535405 (+ 3037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:47.536589 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:47.536600 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:47.537537 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:47.538089 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:47.538120 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:47.539890 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:47.539899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:47.542019 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:47.542042 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:47.543071 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:47.543085 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:47.543313 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:47.544104 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:47.544141 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:47.549733 (+ 5592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:47.555117 (+ 5384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:47.555132 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:47.555145 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:47.555742 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:47.556422 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:47.556426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:47.556429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:47.556676 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:47.558009 (+ 1333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:47.558017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:47.558273 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 05:11:47.558526 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 05:11:47.559028 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":423,"threads_started":1}
I20260927 05:11:47.563689 22883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:11:47.565120 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.564534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39286)
0927 05:11:47.564808 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.564949 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 05:11:47.565469 22833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
I20260927 05:11:47.567217 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.566751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39298)
0927 05:11:47.566940 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.567036 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 05:11:47.567436 22883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39715: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 05:11:47.583096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39715
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44343,127.0.97.61:39715,127.0.97.60:36689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:11:47.616833 22883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } attempt: 1
I20260927 05:11:47.618057 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.617550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39306)
0927 05:11:47.617780 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.617887 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 05:11:47.619088 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.618699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39314)
0927 05:11:47.618841 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.618929 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:11:47.619275 22883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39715: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
I20260927 05:11:47.687551 22883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } attempt: 2
I20260927 05:11:47.688871 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.688295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39316)
0927 05:11:47.688575 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.688697 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 05:11:47.689977 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.689571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39326)
0927 05:11:47.689730 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.689824 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 05:11:47.690181 22883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39715: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
I20260927 05:11:47.781494 22883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } attempt: 3
I20260927 05:11:47.782936 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.782279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39342)
0927 05:11:47.782554 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.782714 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 05:11:47.784221 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.783740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39348)
0927 05:11:47.783911 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.784030 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 05:11:47.784497 22883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39715: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
W20260927 05:11:47.918763 22890 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:11:47.919145 22890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:47.919268 22890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:47.919411 22890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:47.926594 22890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:47.926685 22890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:47.926795 22890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:47.926865 22890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:47.926973 22890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:47.927778 22883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } attempt: 4
I20260927 05:11:47.929026 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.928411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39354)
0927 05:11:47.928740 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.928865 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260927 05:11:47.930135 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:47.929712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39356)
0927 05:11:47.929878 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:47.929977 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 05:11:47.930356 22883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39715: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39715: connect: Connection refused (error 111)
I20260927 05:11:47.937657 22890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44343,127.0.97.61:39715,127.0.97.60:36689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:47.938961 22890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:47.940325 22890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:47.952100 22898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:47.952157 22896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:47.952643 22895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:47.952913 22890 server_base.cc:1034] running on GCE node
I20260927 05:11:47.953727 22890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:47.955060 22890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:47.956300 22890 hybrid_clock.cc:648] HybridClock initialized: now 1790485907956234 us; error 44 us; skew 500 ppm
I20260927 05:11:47.956663 22890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:47 dist-test-slave-hgs0 krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:47.959774 22890 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 05:11:47.961179 22890 webserver.cc:460] Webserver started at http://127.0.97.61:44209/ using document root <none> and password file <none>
I20260927 05:11:47.961898 22890 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:47.962029 22890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:47.962342 22890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:47.965281 22890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/data/instance:
uuid: "9cf709637cab4f609a36e77c9f5c5f3e"
format_stamp: "Formatted at 2026-09-27 05:11:47 on dist-test-slave-hgs0"
server_key: "1171a223f2176b94fbba847779f3ec2c"
server_key_iv: "fde2b5f5e11a49f2f194acb431788227"
server_key_version: "encryptionkey@0"
I20260927 05:11:47.966065 22890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/wal/instance:
uuid: "9cf709637cab4f609a36e77c9f5c5f3e"
format_stamp: "Formatted at 2026-09-27 05:11:47 on dist-test-slave-hgs0"
server_key: "1171a223f2176b94fbba847779f3ec2c"
server_key_iv: "fde2b5f5e11a49f2f194acb431788227"
server_key_version: "encryptionkey@0"
I20260927 05:11:47.971305 22890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 05:11:47.974712 22905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:47.976053 22890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 05:11:47.976238 22890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/wal
uuid: "9cf709637cab4f609a36e77c9f5c5f3e"
format_stamp: "Formatted at 2026-09-27 05:11:47 on dist-test-slave-hgs0"
server_key: "1171a223f2176b94fbba847779f3ec2c"
server_key_iv: "fde2b5f5e11a49f2f194acb431788227"
server_key_version: "encryptionkey@0"
I20260927 05:11:47.976457 22890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:47.992770 22890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:47.997464 22890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:47.997783 22890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:48.017554 22890 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39715
I20260927 05:11:48.017580 22956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39715 every 8 connection(s)
I20260927 05:11:48.019078 22890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
I20260927 05:11:48.022756 22957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:48.028915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 22890
I20260927 05:11:48.029148 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-1/wal/instance
I20260927 05:11:48.029595 388 external_mini_cluster.cc:1428] Setting key 3b5b8809d83d41bed190ae5d53d9c606
I20260927 05:11:48.029256 22957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:48.052245 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.032042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.0.97.62:44343)
0927 05:11:48.032277 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.032286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.033488 (+ 1202us) server_negotiation.cc:407] Connection header received
0927 05:11:48.035597 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.035607 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.035710 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.035839 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.037501 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.038270 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.040832 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.041314 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.041425 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.045809 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.045815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.045818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.045855 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.048838 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.049680 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.049710 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.049715 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.049899 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.050359 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.050364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.050368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.050666 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.050839 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.051575 (+ 736us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.051846 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:11:48.053292 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.032446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44343 (local address 127.0.0.1:52190)
0927 05:11:48.033283 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.033330 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:48.035233 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:48.036166 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:48.036181 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:48.036712 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:48.037288 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.037305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.038448 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.038456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.040620 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.040671 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.041563 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.041572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.041886 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.042684 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:48.042718 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:48.045540 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:48.049012 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.049023 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.049042 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.049525 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.050019 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.050024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.050028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.050235 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.050807 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:48.050817 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:48.051196 (+ 379us) client_negotiation.cc:769] Sending connection context
0927 05:11:48.051475 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 05:11:48.052068 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260927 05:11:48.055269 22957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:11:48.072526 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.056197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39360)
0927 05:11:48.056397 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.056425 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:48.056523 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:48.057638 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:48.057643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:48.058186 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:48.058341 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.058350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.059852 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.059860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.061377 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.061395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.062252 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.062261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.062477 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.063121 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:48.063157 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:48.065803 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:48.069980 (+ 4177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.069985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.069987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.070368 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.070838 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.070841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.070843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.070970 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.071724 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:48.071727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:48.071872 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 05:11:48.072051 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 05:11:48.072268 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 05:11:48.073037 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.056519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.0.97.61:39715)
0927 05:11:48.057133 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.057139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.057166 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:11:48.057243 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.057254 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.057494 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.057668 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.058545 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.059684 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.061524 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.062085 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.062183 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.066036 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.066050 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.066065 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.066096 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.069806 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.070532 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.070543 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.070549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.070727 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.071087 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.071094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.071100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.071593 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.071835 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.072208 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.072495 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":343,"threads_started":1}
I20260927 05:11:48.075371 22957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:11:48.076503 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.076045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:34994)
0927 05:11:48.076224 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.076342 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 05:11:48.076803 22906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
I20260927 05:11:48.078464 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.078140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35002)
0927 05:11:48.078282 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.078365 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 05:11:48.078672 22957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 05:11:48.092511 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36689
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44343,127.0.97.61:39715,127.0.97.60:36689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:11:48.128058 22957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } attempt: 1
I20260927 05:11:48.129216 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.128757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35008)
0927 05:11:48.128957 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.129060 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 05:11:48.130195 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.129852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35014)
0927 05:11:48.130004 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.130097 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:11:48.130421 22957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
I20260927 05:11:48.198689 22957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } attempt: 2
I20260927 05:11:48.199841 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.199379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35030)
0927 05:11:48.199591 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.199690 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 05:11:48.201094 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.200741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35032)
0927 05:11:48.200893 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.200972 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 05:11:48.201406 22957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
I20260927 05:11:48.229626 22883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } attempt: 5
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:11:48.244671 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.230653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.0.97.61:39715)
0927 05:11:48.230829 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.230834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.230853 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:11:48.230907 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.230911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.230987 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.231073 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.232180 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.232952 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.234708 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.235129 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.235230 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.239225 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.239231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.239234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.239268 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.241929 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.242705 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.242709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.242711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.242826 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.243237 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.243241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.243243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.243652 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.243769 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.244271 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.244442 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:11:48.244784 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.230365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39374)
0927 05:11:48.230605 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.230628 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:48.230774 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:48.231161 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:48.231166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:48.231681 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:48.231929 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.231970 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.233105 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.233110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.234514 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.234530 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.235303 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.235313 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.235555 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.236428 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:48.236454 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:48.239001 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:48.242090 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.242095 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.242099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.242593 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.242951 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.242955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.242959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.243129 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.243774 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:48.243779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:48.243971 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 05:11:48.244155 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 05:11:48.244441 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 05:11:48.245813 22883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:11:48.246835 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.246459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35038)
0927 05:11:48.246629 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.246701 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 05:11:48.247772 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.247479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35040)
0927 05:11:48.247614 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.247675 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:11:48.248109 22883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
I20260927 05:11:48.292713 22957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } attempt: 3
I20260927 05:11:48.294092 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.293493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35056)
0927 05:11:48.293746 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.293907 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 05:11:48.295171 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.294767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35070)
0927 05:11:48.294993 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.295065 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260927 05:11:48.295439 22957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
I20260927 05:11:48.299350 22883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } attempt: 1
I20260927 05:11:48.300508 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.300099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35080)
0927 05:11:48.300275 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.300357 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 05:11:48.301510 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.301176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35090)
0927 05:11:48.301333 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.301407 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 05:11:48.301730 22883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
I20260927 05:11:48.369978 22883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } attempt: 2
I20260927 05:11:48.371182 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.370698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35104)
0927 05:11:48.370899 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.371005 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 05:11:48.372188 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.371809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35112)
0927 05:11:48.372000 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.372072 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 05:11:48.372395 22883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
W20260927 05:11:48.434159 22964 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:11:48.434495 22964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:48.434603 22964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:48.434746 22964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 05:11:48.438674 22957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } attempt: 4
I20260927 05:11:48.439855 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.439403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35114)
0927 05:11:48.439625 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.439709 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 05:11:48.440779 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.440493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35130)
0927 05:11:48.440620 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.440682 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 05:11:48.441052 22957 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
W20260927 05:11:48.442880 22964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:48.442961 22964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:48.443109 22964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:48.443183 22964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:48.443292 22964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:48.453454 22964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44343,127.0.97.61:39715,127.0.97.60:36689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:48.454808 22964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:48.456198 22964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:48.472734 22969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:48.473111 22970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:48.472755 22972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:48.473698 22964 server_base.cc:1034] running on GCE node
I20260927 05:11:48.474648 22964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:48.476179 22964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:48.477428 22964 hybrid_clock.cc:648] HybridClock initialized: now 1790485908477346 us; error 63 us; skew 500 ppm
I20260927 05:11:48.477833 22964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 05:11:48.478632 22883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } attempt: 3
I20260927 05:11:48.479774 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.479234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35138)
0927 05:11:48.479483 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.479584 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 05:11:48.480928 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.480515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35150)
0927 05:11:48.480733 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.480829 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 05:11:48.481132 22883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36689: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36689: connect: Connection refused (error 111)
I20260927 05:11:48.481715 22964 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 05:11:48.483181 22964 webserver.cc:460] Webserver started at http://127.0.97.60:41659/ using document root <none> and password file <none>
I20260927 05:11:48.484193 22964 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:48.484325 22964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:48.484668 22964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:48.487563 22964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/data/instance:
uuid: "5ae0589cf49f42e09d934a8697164b98"
format_stamp: "Formatted at 2026-09-27 05:11:48 on dist-test-slave-hgs0"
server_key: "95e3fffd489db9cb0e6ef73cc0e1ae4c"
server_key_iv: "dbbfefa6d2e05d6c0758ed5372929e0c"
server_key_version: "encryptionkey@0"
I20260927 05:11:48.488368 22964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/wal/instance:
uuid: "5ae0589cf49f42e09d934a8697164b98"
format_stamp: "Formatted at 2026-09-27 05:11:48 on dist-test-slave-hgs0"
server_key: "95e3fffd489db9cb0e6ef73cc0e1ae4c"
server_key_iv: "dbbfefa6d2e05d6c0758ed5372929e0c"
server_key_version: "encryptionkey@0"
I20260927 05:11:48.493434 22964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 05:11:48.496876 22979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:48.498275 22964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 05:11:48.498430 22964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/wal
uuid: "5ae0589cf49f42e09d934a8697164b98"
format_stamp: "Formatted at 2026-09-27 05:11:48 on dist-test-slave-hgs0"
server_key: "95e3fffd489db9cb0e6ef73cc0e1ae4c"
server_key_iv: "dbbfefa6d2e05d6c0758ed5372929e0c"
server_key_version: "encryptionkey@0"
I20260927 05:11:48.498636 22964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:48.522734 22964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:48.527521 22964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:48.527770 22964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:48.547166 22964 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36689
I20260927 05:11:48.547112 23030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36689 every 8 connection(s)
I20260927 05:11:48.548753 22964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
I20260927 05:11:48.549129 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 22964
I20260927 05:11:48.549333 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790485535376807-388-0/minicluster-data/master-2/wal/instance
I20260927 05:11:48.549890 388 external_mini_cluster.cc:1428] Setting key bfc9d5d762b793e12444dd16eacb8466
I20260927 05:11:48.553659 23031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:48.561264 23031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:48.577689 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.551868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.0.97.60:36689)
0927 05:11:48.552657 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.552671 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.552717 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:11:48.554493 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.554519 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.554937 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.555376 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.557015 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.558444 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.559859 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.560739 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.560975 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.569588 (+ 8613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.569601 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.569616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.569664 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.573050 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.573886 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.573902 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.573909 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.574071 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.574472 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.574478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.574481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.575177 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.575507 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.576023 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.576488 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":317,"threads_started":1}
I20260927 05:11:48.580016 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.564440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.0.97.62:44343)
0927 05:11:48.565052 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.565058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.565390 (+ 332us) server_negotiation.cc:407] Connection header received
0927 05:11:48.565686 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.565690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.565760 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.565881 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.567146 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.568032 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.569514 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.570204 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.570358 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.574004 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.574010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.574014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.574050 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.577050 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.577777 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.577785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.577789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.577974 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.578414 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.578423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.578426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.578691 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.578857 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.579425 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.579672 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":291,"threads_started":1}
I20260927 05:11:48.580101 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.564373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44343 (local address 127.0.0.1:52202)
0927 05:11:48.565242 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.565282 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:48.565492 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:48.565955 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:48.565963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:48.566676 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:48.566965 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.566988 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.568218 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.568223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.569367 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.569380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.570464 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.570472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.570720 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.571401 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:48.571429 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:48.573778 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:48.577208 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.577219 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.577227 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.577639 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.578114 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.578118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.578121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.578262 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.578830 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:48.578838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:48.579079 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 05:11:48.579317 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 05:11:48.579635 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":367,"threads_started":1}
I20260927 05:11:48.582063 23031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:11:48.597477 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.582894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39715 (local address 127.0.0.1:39378)
0927 05:11:48.583232 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.583252 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:48.583370 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:48.583813 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:48.583818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:48.584296 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:48.584481 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.584491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.585867 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.585871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.587138 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.587154 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.588210 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.588215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.588347 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.588974 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:48.589006 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:48.591632 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:48.594703 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.594709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.594715 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.595217 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.595663 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.595667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.595670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.595856 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.596498 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:48.596505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:48.596670 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 05:11:48.596825 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 05:11:48.597109 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260927 05:11:48.597554 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.582898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.0.97.61:39715)
0927 05:11:48.583082 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.583087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.583341 (+ 254us) server_negotiation.cc:407] Connection header received
0927 05:11:48.583581 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.583587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.583671 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.583751 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.584689 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.585720 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.587321 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.588043 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.588175 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.591836 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.591841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.591844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.591886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.594538 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.595341 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.595351 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.595356 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.595517 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.595998 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.596003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.596005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.596375 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.596494 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.596941 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.597176 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:11:48.597855 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.581211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.0.97.62:44343)
0927 05:11:48.581376 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.581381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.581436 (+ 55us) server_negotiation.cc:407] Connection header received
0927 05:11:48.581612 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.581617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.581710 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.581810 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.583410 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.584474 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.586223 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.586874 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.587011 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.591593 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.591602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.591607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.591647 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.594904 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.595681 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.595690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.595693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.595831 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.596283 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.596286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.596288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.596623 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.596908 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.597262 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.597489 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:11:48.598795 23031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:11:48.613749 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.599744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.0.97.60:36689)
0927 05:11:48.599926 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.599932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.599978 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:11:48.600022 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.600026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.600095 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.600201 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.600913 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.601923 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.603596 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.604310 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.604432 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.607816 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.607825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.607831 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.607872 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.611026 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.611680 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.611688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.611691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.611820 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.612226 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.612234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.612238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.612524 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.612672 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.613023 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.613291 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 05:11:48.613750 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.599514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35180)
0927 05:11:48.599741 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.599759 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:48.599857 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:48.600251 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:48.600257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:48.600641 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:48.600771 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.600779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.602034 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.602040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.603413 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.603424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.604487 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.604493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.604667 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.605335 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:48.605360 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:48.607596 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:48.611183 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.611188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.611192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.611560 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.611965 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.611971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.611974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.612118 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.612649 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:48.612654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:48.612809 (+ 155us) client_negotiation.cc:769] Sending connection context
0927 05:11:48.612950 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 05:11:48.613449 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":175}
I20260927 05:11:48.616285 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.599589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.0.97.61:39715)
0927 05:11:48.599920 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.599926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.599972 (+ 46us) server_negotiation.cc:407] Connection header received
0927 05:11:48.600018 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.600023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.600087 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.600183 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.601610 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.602719 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.604401 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.605065 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.605201 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.610328 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.610334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.610337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.610375 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.613112 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.614034 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.614040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.614045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.614221 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.614683 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.614690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.614693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.614997 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.615284 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.615561 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.615808 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260927 05:11:48.619820 23031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98: Bootstrap starting.
I20260927 05:11:48.621449 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:48.624069 23031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:48.625377 23031 log.cc:824] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:48.625180 22858 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "62ed78094dd444ac8a3b09802bc6e573"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52208
I20260927 05:11:48.628243 23031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98: No bootstrap required, opened a new log
I20260927 05:11:48.635614 23031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } }
I20260927 05:11:48.636029 23031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae0589cf49f42e09d934a8697164b98, State: Initialized, Role: FOLLOWER
I20260927 05:11:48.636534 23031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } }
I20260927 05:11:48.637532 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } } }
I20260927 05:11:48.637941 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:48.638589 23031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 05:11:48.648468 23049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:11:48.648624 23049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:11:48.646452 22964 master_runner.cc:186] Error getting master registration for 127.0.97.62:44343: OK, instance_id { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" instance_seqno: 1790485907505385 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:11:48.654944 22964 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:11:48.658365 22883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } attempt: 4
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:11:48.675021 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.659194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.0.97.60:36689)
0927 05:11:48.659403 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.659410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.659563 (+ 153us) server_negotiation.cc:407] Connection header received
0927 05:11:48.659773 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.659778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.659861 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.660002 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.661041 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.662250 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.664184 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.664927 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.665069 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.668943 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.668953 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.668961 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.669006 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.672169 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.672907 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.672913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.672919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.673053 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.673463 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.673468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.673472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.673808 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.673976 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.674404 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.674624 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 05:11:48.675016 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.659160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35182)
0927 05:11:48.659458 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.659481 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:48.659623 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:48.660083 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:48.660089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:48.660618 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:48.660873 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.660887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.662431 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.662439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.663982 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.664002 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.665100 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.665109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.665315 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.666014 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:48.666038 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:48.668716 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:48.672342 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.672349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.672354 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.672769 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.673190 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.673195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.673199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.673347 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.673936 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:48.673941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:48.674126 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 05:11:48.674284 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 05:11:48.674624 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 05:11:48.680624 22883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573: Bootstrap starting.
I20260927 05:11:48.685144 22883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:48.686532 22883 log.cc:824] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:48.689576 22883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573: No bootstrap required, opened a new log
I20260927 05:11:48.694912 22883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } }
I20260927 05:11:48.695295 22883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ed78094dd444ac8a3b09802bc6e573, State: Initialized, Role: FOLLOWER
I20260927 05:11:48.695761 22883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } }
I20260927 05:11:48.696762 23052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } } }
I20260927 05:11:48.697191 23052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:48.697964 22883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:48.698115 22858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 05:11:48.698334 22858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:48.702062 22858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } }
I20260927 05:11:48.705083 22858 leader_election.cc:288] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [CANDIDATE]: Term 1 election: Requested vote from peers 9cf709637cab4f609a36e77c9f5c5f3e (127.0.97.61:39715), 5ae0589cf49f42e09d934a8697164b98 (127.0.97.60:36689)
I20260927 05:11:48.705799 23006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed78094dd444ac8a3b09802bc6e573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5ae0589cf49f42e09d934a8697164b98"
I20260927 05:11:48.706307 23006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:48.709965 23006 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62ed78094dd444ac8a3b09802bc6e573 in term 1.
I20260927 05:11:48.710731 22832 leader_election.cc:302] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae0589cf49f42e09d934a8697164b98, 62ed78094dd444ac8a3b09802bc6e573; no voters:
I20260927 05:11:48.711297 23052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:48.713567 23052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [term 1 LEADER]: Becoming Leader. State: Replica: 62ed78094dd444ac8a3b09802bc6e573, State: Running, Role: LEADER
I20260927 05:11:48.714250 23052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } }
I20260927 05:11:48.720158 23064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed78094dd444ac8a3b09802bc6e573. Latest consensus state: current_term: 1 leader_uuid: "62ed78094dd444ac8a3b09802bc6e573" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } } }
I20260927 05:11:48.720562 23064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:48.730438 23067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:48.736425 23067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:48.737831 23063 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 05:11:48.740262 22957 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } attempt: 5
W20260927 05:11:48.744679 22837 tablet.cc:2369] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 05:11:48 dist-test-slave-hgs0 krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:11:48.756454 23006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [term 1 FOLLOWER]: Refusing update from remote peer 62ed78094dd444ac8a3b09802bc6e573: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:11:48.758265 23052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:11:48.742811 22816 master_runner.cc:186] Error getting master registration for 127.0.97.61:39715: OK, instance_id { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" instance_seqno: 1790485908012020 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:11:48.758708 22816 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:11:48.769165 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.741616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.0.97.60:36689)
0927 05:11:48.741803 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.741810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.741829 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:11:48.741890 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.741895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.741983 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.742086 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.743242 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.744397 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.746244 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.746883 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.747012 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.753186 (+ 6174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.753193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.753198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.753246 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.757446 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.766033 (+ 8587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.766040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.766044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.766255 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.766975 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.766983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.766990 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.767401 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.767539 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.768432 (+ 893us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.768711 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:11:48.772696 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.740836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36689 (local address 127.0.0.1:35192)
0927 05:11:48.741217 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:48.741239 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:48.741371 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:48.742269 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:48.742280 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:48.742833 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:48.743051 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.743063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.744557 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.744562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.746039 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:48.746056 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.747220 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.747227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:48.747451 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.748326 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:48.748387 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:48.752861 (+ 4474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:48.758365 (+ 5504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.758371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.758395 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.765787 (+ 7392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.766495 (+ 708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:48.766500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:48.766504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:48.766777 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:48.767703 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:48.767709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:48.767985 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 05:11:48.769793 (+ 1808us) client_negotiation.cc:240] Negotiation successful
0927 05:11:48.770374 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
W20260927 05:11:48.780311 22985 tablet.cc:2369] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:11:48.789425 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.728317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.0.97.61:39715)
0927 05:11:48.728581 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.728591 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.760030 (+ 31439us) server_negotiation.cc:407] Connection header received
0927 05:11:48.761401 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.761406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.761490 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.761624 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.765797 (+ 4173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.766930 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.770135 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.770897 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.771036 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.779032 (+ 7996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.779040 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.779044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.779094 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.784271 (+ 5177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.785924 (+ 1653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.785930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.785934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.786106 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.787008 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.787015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.787018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.787429 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.787734 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.788754 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.789028 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 05:11:48.791563 22957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e: Bootstrap starting.
I20260927 05:11:48.797084 22957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:48.801281 22957 log.cc:824] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:48.804715 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed78094dd444ac8a3b09802bc6e573. Latest consensus state: current_term: 1 leader_uuid: "62ed78094dd444ac8a3b09802bc6e573" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } } }
I20260927 05:11:48.805063 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:48.807577 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.760510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35198 (local address 127.0.97.60:36689)
0927 05:11:48.769467 (+ 8957us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.769476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.776375 (+ 6899us) server_negotiation.cc:407] Connection header received
0927 05:11:48.777524 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.777530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.777636 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.777716 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.784818 (+ 7102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.785872 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.787773 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.788478 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.788586 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.798477 (+ 9891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.798484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.798489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.798539 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.802755 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.803868 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.803875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.803879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.804084 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.804585 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.804590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.804594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.805004 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.805254 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.806110 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.806686 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8819,"thread_start_us":23422,"threads_started":1}
I20260927 05:11:48.812136 23064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62ed78094dd444ac8a3b09802bc6e573" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } } }
I20260927 05:11:48.812496 23064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:48.818701 22957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e: No bootstrap required, opened a new log
I20260927 05:11:48.827556 23067 catalog_manager.cc:1347] Generated new cluster ID: 572d9d3696914fd285e6d6c34d63ffb6
I20260927 05:11:48.827678 23067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:48.830835 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:48.729873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52224 (local address 127.0.97.62:44343)
0927 05:11:48.730080 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:48.730086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:48.762044 (+ 31958us) server_negotiation.cc:407] Connection header received
0927 05:11:48.762264 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:48.762270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:48.762360 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:48.762466 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:48.784031 (+ 21565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.785303 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.806631 (+ 21328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:48.807407 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:48.807536 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:48.822915 (+ 15379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:48.822924 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:48.822930 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:48.823018 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:48.827086 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.828174 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.828185 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.828189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.828365 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:48.828843 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:48.828848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:48.828852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:48.829283 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:48.829568 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:48.830166 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 05:11:48.830436 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 05:11:48.831475 22957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } }
I20260927 05:11:48.831873 22957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf709637cab4f609a36e77c9f5c5f3e, State: Initialized, Role: FOLLOWER
I20260927 05:11:48.832451 22957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } }
I20260927 05:11:48.841886 22957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:48.842664 22932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62ed78094dd444ac8a3b09802bc6e573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9cf709637cab4f609a36e77c9f5c5f3e"
I20260927 05:11:48.843441 22931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:48.843346 23082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } } }
I20260927 05:11:48.843801 23082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:48.848824 22931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [term 1 FOLLOWER]: Refusing update from remote peer 62ed78094dd444ac8a3b09802bc6e573: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:11:48.854807 22932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62ed78094dd444ac8a3b09802bc6e573 in term 1.
I20260927 05:11:48.857442 23052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:11:48.866371 23038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62ed78094dd444ac8a3b09802bc6e573" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } } }
I20260927 05:11:48.866652 23038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:48.875659 23067 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:48.878185 23067 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:48.910194 23067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573: Generated new TSK 0
I20260927 05:11:48.911710 23067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:48.916780 23082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ed78094dd444ac8a3b09802bc6e573. Latest consensus state: current_term: 1 leader_uuid: "62ed78094dd444ac8a3b09802bc6e573" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } } }
I20260927 05:11:48.917057 23082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:48.917587 23093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62ed78094dd444ac8a3b09802bc6e573" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } } }
I20260927 05:11:48.917994 23093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:48.918020 23052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62ed78094dd444ac8a3b09802bc6e573" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62ed78094dd444ac8a3b09802bc6e573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44343 } } peers { permanent_uuid: "9cf709637cab4f609a36e77c9f5c5f3e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39715 } } peers { permanent_uuid: "5ae0589cf49f42e09d934a8697164b98" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36689 } } }
I20260927 05:11:48.918394 23052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ed78094dd444ac8a3b09802bc6e573 [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:48.959291 23097 catalog_manager.cc:1259] Loaded cluster ID: 572d9d3696914fd285e6d6c34d63ffb6
I20260927 05:11:48.959463 23097 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e: loading cluster ID for follower catalog manager: success
I20260927 05:11:48.963423 23097 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e: acquiring CA information for follower catalog manager: success
I20260927 05:11:48.965937 23097 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9cf709637cab4f609a36e77c9f5c5f3e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 05:11:49.054898 23036 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.040640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.0.97.62:44343)
0927 05:11:49.040839 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.040846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.052734 (+ 11888us) server_negotiation.cc:407] Connection header received
0927 05:11:49.053053 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.053059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.053150 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.053228 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.054617 (+ 1389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52228: BlockingRecv error: recv got EOF from 127.0.0.1:52228 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 05:11:49.057917 22961 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.048103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.0.97.61:39715)
0927 05:11:49.048394 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.048403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.055853 (+ 7450us) server_negotiation.cc:407] Connection header received
0927 05:11:49.056248 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.056255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.056347 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.056433 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.057665 (+ 1232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39402: BlockingRecv error: recv got EOF from 127.0.0.1:39402 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 05:11:49.065285 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.052976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.0.97.60:36689)
0927 05:11:49.053248 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.053258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.058680 (+ 5422us) server_negotiation.cc:407] Connection header received
0927 05:11:49.058882 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.058889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.058976 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.059054 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.065012 (+ 5958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35214: BlockingRecv error: recv got EOF from 127.0.0.1:35214 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 05:11:49.117281 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.097937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.0.97.61:39715)
0927 05:11:49.098147 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.098155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.109004 (+ 10849us) server_negotiation.cc:407] Connection header received
0927 05:11:49.109587 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.109594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.109815 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.109948 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.110644 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.111739 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.113800 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.114629 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.114734 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.115571 (+ 837us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.116403 (+ 832us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.116537 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.116794 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.116915 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 05:11:49.163805 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.150757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.0.97.60:36689)
0927 05:11:49.150948 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.150954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.161675 (+ 10721us) server_negotiation.cc:407] Connection header received
0927 05:11:49.162254 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.162260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.162387 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.162464 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.163519 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35220: BlockingRecv error: recv got EOF from 127.0.0.1:35220 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 05:11:49.223977 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.206053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.0.97.61:39715)
0927 05:11:49.206272 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.206280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.216921 (+ 10641us) server_negotiation.cc:407] Connection header received
0927 05:11:49.217463 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.217469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.217575 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.217665 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.218335 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.219474 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.221531 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.222248 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.222348 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.223036 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.223210 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.223275 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.223499 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.223592 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 05:11:49.272065 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.258234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.0.97.60:36689)
0927 05:11:49.258486 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.258494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.269288 (+ 10794us) server_negotiation.cc:407] Connection header received
0927 05:11:49.269997 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.270008 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.270165 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.270264 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.271680 (+ 1416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35230: BlockingRecv error: recv got EOF from 127.0.0.1:35230 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:11:49.332156 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.313976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.0.97.61:39715)
0927 05:11:49.314216 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.314223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.324600 (+ 10377us) server_negotiation.cc:407] Connection header received
0927 05:11:49.325210 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.325218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.325320 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.325398 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.326199 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.327347 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.329578 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.330338 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.330460 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.331187 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.331341 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.331412 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.331664 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.331762 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 05:11:49.377923 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.364856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.0.97.60:36689)
0927 05:11:49.365059 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.365068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.375713 (+ 10645us) server_negotiation.cc:407] Connection header received
0927 05:11:49.376307 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.376314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.376420 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.376504 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.377645 (+ 1141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35234: BlockingRecv error: recv got EOF from 127.0.0.1:35234 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:11:49.394162 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.387031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.0.97.61:39715)
0927 05:11:49.387293 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.387299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.387855 (+ 556us) server_negotiation.cc:407] Connection header received
0927 05:11:49.388071 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.388079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.388189 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.388304 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.388775 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.389985 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.391973 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.392685 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.392824 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.393328 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.393471 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.393572 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.393751 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.393844 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
W20260927 05:11:49.403811 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.401635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.0.97.60:36689)
0927 05:11:49.401930 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.401937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.401961 (+ 24us) server_negotiation.cc:407] Connection header received
0927 05:11:49.402309 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.402315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.402415 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.402534 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.403566 (+ 1032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35250: BlockingRecv error: recv got EOF from 127.0.0.1:35250 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
I20260927 05:11:49.424204 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.416199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.0.97.61:39715)
0927 05:11:49.416396 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.416403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.416941 (+ 538us) server_negotiation.cc:407] Connection header received
0927 05:11:49.417293 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.417299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.417387 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.417459 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.418570 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.419666 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.421827 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.422465 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.422569 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.423252 (+ 683us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.423404 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.423491 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.423747 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.423834 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 05:11:49.440728 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.438322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.0.97.60:36689)
0927 05:11:49.438521 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.438527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.439085 (+ 558us) server_negotiation.cc:407] Connection header received
0927 05:11:49.439379 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.439388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.439490 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.439597 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.440490 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35266: BlockingRecv error: recv got EOF from 127.0.0.1:35266 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 05:11:49.468255 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.460857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.0.97.61:39715)
0927 05:11:49.461071 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.461077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.461652 (+ 575us) server_negotiation.cc:407] Connection header received
0927 05:11:49.461901 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.461910 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.462028 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.462159 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.462709 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.463897 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.466000 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.466678 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.466811 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.467413 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.467563 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.467658 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.467841 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.467979 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 05:11:49.478188 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.475566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.0.97.60:36689)
0927 05:11:49.475739 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.475743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.476506 (+ 763us) server_negotiation.cc:407] Connection header received
0927 05:11:49.476764 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.476768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.476855 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.476973 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.477911 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35278: BlockingRecv error: recv got EOF from 127.0.0.1:35278 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 05:11:49.529372 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.522083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.0.97.61:39715)
0927 05:11:49.522321 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.522328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.522954 (+ 626us) server_negotiation.cc:407] Connection header received
0927 05:11:49.523152 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.523158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.523274 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.523379 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.523843 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.525039 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.527117 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.527811 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.527978 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.528518 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.528688 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.528780 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.528961 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.529048 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 05:11:49.542783 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.540452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.97.60:36689)
0927 05:11:49.540807 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.540814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.541124 (+ 310us) server_negotiation.cc:407] Connection header received
0927 05:11:49.541357 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.541361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.541439 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.541531 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.542499 (+ 968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35290: BlockingRecv error: recv got EOF from 127.0.0.1:35290 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
I20260927 05:11:49.616099 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.610752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.0.97.61:39715)
0927 05:11:49.611010 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.611020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.611292 (+ 272us) server_negotiation.cc:407] Connection header received
0927 05:11:49.611545 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.611550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.611629 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.611734 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.612204 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.612931 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.614286 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.614742 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.614889 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.615304 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.615425 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.615528 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.615691 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.615780 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260927 05:11:49.623934 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:49.622125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.0.97.60:36689)
0927 05:11:49.622336 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.622341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.622663 (+ 322us) server_negotiation.cc:407] Connection header received
0927 05:11:49.622814 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.622817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.622902 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.622993 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:49.623685 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35294: BlockingRecv error: recv got EOF from 127.0.0.1:35294 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:11:49.650645 23049 catalog_manager.cc:1259] Loaded cluster ID: 572d9d3696914fd285e6d6c34d63ffb6
I20260927 05:11:49.650748 23049 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98: loading cluster ID for follower catalog manager: success
I20260927 05:11:49.653389 23049 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98: acquiring CA information for follower catalog manager: success
I20260927 05:11:49.655222 23049 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ae0589cf49f42e09d934a8697164b98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:11:49.760988 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.755733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.0.97.61:39715)
0927 05:11:49.756017 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.756024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.756350 (+ 326us) server_negotiation.cc:407] Connection header received
0927 05:11:49.756582 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.756589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.756668 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.756777 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.757230 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.757936 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.759278 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.759702 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.759843 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.760274 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.760384 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.760483 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.760638 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.760717 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 05:11:49.773204 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.767358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.0.97.60:36689)
0927 05:11:49.767642 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.767648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.767888 (+ 240us) server_negotiation.cc:407] Connection header received
0927 05:11:49.768123 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.768128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.768280 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.768380 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.768789 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.769487 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.770912 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.771331 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.771429 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.771989 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.772542 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.772683 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.772815 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.772883 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 05:11:49.788110 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.779810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.0.97.62:44343)
0927 05:11:49.780569 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.780580 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.780856 (+ 276us) server_negotiation.cc:407] Connection header received
0927 05:11:49.781216 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.781226 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.781474 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.781644 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.782593 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.783468 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.785750 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.786190 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.786335 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.786990 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.787354 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.787481 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.787616 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.787709 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":405,"threads_started":1}
I20260927 05:11:49.789225 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.781049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.0.97.60:36689)
0927 05:11:49.781280 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.781286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.782606 (+ 1320us) server_negotiation.cc:407] Connection header received
0927 05:11:49.782894 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.782900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.783022 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.783092 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.784082 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.784932 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.786830 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.787332 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.787487 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.788039 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.788180 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.788282 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.788794 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.788884 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 05:11:49.791430 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:49.780001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.0.97.61:39715)
0927 05:11:49.780260 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:49.780267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:49.782331 (+ 2064us) server_negotiation.cc:407] Connection header received
0927 05:11:49.782677 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:49.782685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:49.782776 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:49.782844 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:49.783776 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.784970 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.789324 (+ 4354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:49.789981 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:49.790110 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:49.790709 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:49.790852 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:49.790949 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:49.791083 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 05:11:49.791170 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 05:11:49.793617 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22816
I20260927 05:11:49.794883 22878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:49.905599 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 22816
W20260927 05:11:49.926373 22908 connection.cc:535] server connection from 127.0.0.1:39374 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 05:11:49.926723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22890
I20260927 05:11:49.927819 22952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:50.036854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 22890
I20260927 05:11:50.057428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 22964
I20260927 05:11:50.058693 23026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:50.166272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 22964
2026-09-27T05:11:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4524 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 05:11:50 dist-test-slave-hgs0 krb5kdc[23212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 05:11:50 dist-test-slave-hgs0 krb5kdc[23212](info): set up 2 sockets
Sep 27 05:11:50 dist-test-slave-hgs0 krb5kdc[23212](info): commencing operation
krb5kdc: starting...
W20260927 05:11:51.304497 388 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/test-user.keytab.
Sep 27 05:11:51 dist-test-slave-hgs0 krb5kdc[23212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T05:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T05:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260927 05:11:51.580327 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34707
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34707,127.0.97.61:35701,127.0.97.60:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 05:11:51.925711 23225 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:11:51.926090 23225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:51.926204 23225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:51.926327 23225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:51.933768 23225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:51.933864 23225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:51.933967 23225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:51.934062 23225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:51.934170 23225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:51.944969 23225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34707,127.0.97.61:35701,127.0.97.60:43235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:51.946354 23225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:51.947750 23225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:51.959890 23231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:51.960067 23233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:51.960108 23230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:51.960686 23225 server_base.cc:1034] running on GCE node
I20260927 05:11:51.961541 23225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:51.962944 23225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:51.964179 23225 hybrid_clock.cc:648] HybridClock initialized: now 1790485911964123 us; error 31 us; skew 500 ppm
I20260927 05:11:51.964502 23225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:51 dist-test-slave-hgs0 krb5kdc[23212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:51.967655 23225 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260927 05:11:51.969375 23225 webserver.cc:460] Webserver started at http://127.0.97.62:45491/ using document root <none> and password file <none>
I20260927 05:11:51.970126 23225 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:51.970263 23225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:51.970603 23225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:51.973698 23225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/data/instance:
uuid: "6de2d1b3e6484b77b37232953392b65e"
format_stamp: "Formatted at 2026-09-27 05:11:51 on dist-test-slave-hgs0"
server_key: "cf1f33fb03892574f7cbead4b1360310"
server_key_iv: "c6b87df19b75db3937a1513fd20625da"
server_key_version: "encryptionkey@0"
I20260927 05:11:51.974515 23225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/wal/instance:
uuid: "6de2d1b3e6484b77b37232953392b65e"
format_stamp: "Formatted at 2026-09-27 05:11:51 on dist-test-slave-hgs0"
server_key: "cf1f33fb03892574f7cbead4b1360310"
server_key_iv: "c6b87df19b75db3937a1513fd20625da"
server_key_version: "encryptionkey@0"
I20260927 05:11:51.979902 23225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 05:11:51.983502 23240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:51.984953 23225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 05:11:51.985155 23225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/wal
uuid: "6de2d1b3e6484b77b37232953392b65e"
format_stamp: "Formatted at 2026-09-27 05:11:51 on dist-test-slave-hgs0"
server_key: "cf1f33fb03892574f7cbead4b1360310"
server_key_iv: "c6b87df19b75db3937a1513fd20625da"
server_key_version: "encryptionkey@0"
I20260927 05:11:51.985386 23225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:52.016834 23225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:52.021226 23225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:52.021513 23225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:52.040494 23225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34707
I20260927 05:11:52.040508 23291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34707 every 8 connection(s)
I20260927 05:11:52.041875 23225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/data/info.pb
I20260927 05:11:52.045543 23292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:52.049464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 23225
I20260927 05:11:52.049660 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-0/wal/instance
I20260927 05:11:52.050187 388 external_mini_cluster.cc:1428] Setting key e53519d129a30f5edde1c0fe9b1c293a
I20260927 05:11:52.053985 23292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:11:52 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260927 05:11:52.080471 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.057920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34707 (local address 127.0.0.1:56766)
0927 05:11:52.059122 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.059176 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:52.061524 (+ 2348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:52.062534 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:52.062539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:52.063144 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:52.063479 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:52.063495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.065835 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.065839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:52.066938 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:52.066950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.068316 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.068321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:52.068440 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:52.069143 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:52.069165 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:52.071377 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:52.076165 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:52.076176 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:52.076186 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:52.076583 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:52.077055 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:52.077058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:52.077060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:52.077218 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:52.078293 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:52.078299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:52.078541 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 05:11:52.078833 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 05:11:52.079255 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"spinlock_wait_cycles":32640,"thread_start_us":697,"threads_started":1}
I20260927 05:11:52.080518 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.057921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.0.97.62:34707)
0927 05:11:52.058797 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:52.058816 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:52.059392 (+ 576us) server_negotiation.cc:407] Connection header received
0927 05:11:52.061935 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:52.061946 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:52.062283 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:52.062494 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:52.063695 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.065674 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.067095 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.068100 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.068292 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:52.071631 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:52.071646 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:52.071667 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:52.071711 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:52.076000 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:52.076724 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:52.076733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:52.076737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:52.076935 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:52.077330 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:52.077335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:52.077339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:52.078107 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:52.078327 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:52.079004 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 05:11:52.079401 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":612,"thread_start_us":363,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260927 05:11:52.083591 23292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:11:52.084792 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.084384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41284)
0927 05:11:52.084566 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.084677 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 05:11:52.085121 23243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
I20260927 05:11:52.086742 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.086337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41298)
0927 05:11:52.086535 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.086610 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 05:11:52.086954 23292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35701: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260927 05:11:52.116575 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35701
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34707,127.0.97.61:35701,127.0.97.60:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:11:52.136307 23292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } attempt: 1
I20260927 05:11:52.138034 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.137478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41310)
0927 05:11:52.137742 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.137847 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 05:11:52.139614 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.139001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41320)
0927 05:11:52.139248 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.139341 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 05:11:52.139855 23292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35701: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
I20260927 05:11:52.208230 23292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } attempt: 2
I20260927 05:11:52.209575 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.209055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41328)
0927 05:11:52.209277 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.209385 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 05:11:52.210623 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.210243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41336)
0927 05:11:52.210393 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.210484 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 05:11:52.210811 23292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35701: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
I20260927 05:11:52.302106 23292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } attempt: 3
I20260927 05:11:52.303231 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.302780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41338)
0927 05:11:52.302991 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.303089 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 05:11:52.304374 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.304029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41348)
0927 05:11:52.304176 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.304240 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 05:11:52.304548 23292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35701: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
I20260927 05:11:52.447829 23292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } attempt: 4
I20260927 05:11:52.449064 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.448536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41358)
0927 05:11:52.448848 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.448932 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 05:11:52.450273 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.449900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41372)
0927 05:11:52.450079 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.450148 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 05:11:52.450457 23292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:35701: Network error: Client connection negotiation failed: client connection to 127.0.97.61:35701: connect: Connection refused (error 111)
W20260927 05:11:52.485082 23299 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:11:52.485481 23299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:52.485611 23299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:52.485750 23299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:52.493417 23299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:52.493551 23299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:52.493664 23299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:52.493752 23299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:52.493841 23299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:52.504258 23299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34707,127.0.97.61:35701,127.0.97.60:43235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:52.505721 23299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:52.507159 23299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:52.520087 23304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:52.520040 23307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:52.520154 23305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:52.520990 23299 server_base.cc:1034] running on GCE node
I20260927 05:11:52.522009 23299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:52.523663 23299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:52.524995 23299 hybrid_clock.cc:648] HybridClock initialized: now 1790485912524888 us; error 100 us; skew 500 ppm
I20260927 05:11:52.525410 23299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:52 dist-test-slave-hgs0 krb5kdc[23212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:52.529419 23299 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260927 05:11:52.531193 23299 webserver.cc:460] Webserver started at http://127.0.97.61:40889/ using document root <none> and password file <none>
I20260927 05:11:52.532157 23299 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:52.532302 23299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:52.532666 23299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:52.535904 23299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/data/instance:
uuid: "44717074f0294bb4b465fb549300697d"
format_stamp: "Formatted at 2026-09-27 05:11:52 on dist-test-slave-hgs0"
server_key: "7ae1d81638548ade080b29961fb452f8"
server_key_iv: "4ca6b33839b5b29be193d412c5e389bf"
server_key_version: "encryptionkey@0"
I20260927 05:11:52.536911 23299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/wal/instance:
uuid: "44717074f0294bb4b465fb549300697d"
format_stamp: "Formatted at 2026-09-27 05:11:52 on dist-test-slave-hgs0"
server_key: "7ae1d81638548ade080b29961fb452f8"
server_key_iv: "4ca6b33839b5b29be193d412c5e389bf"
server_key_version: "encryptionkey@0"
I20260927 05:11:52.542752 23299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 05:11:52.546617 23314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:52.548202 23299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:11:52.548408 23299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/wal
uuid: "44717074f0294bb4b465fb549300697d"
format_stamp: "Formatted at 2026-09-27 05:11:52 on dist-test-slave-hgs0"
server_key: "7ae1d81638548ade080b29961fb452f8"
server_key_iv: "4ca6b33839b5b29be193d412c5e389bf"
server_key_version: "encryptionkey@0"
I20260927 05:11:52.548656 23299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:52.574173 23299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:52.579509 23299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:52.579840 23299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:52.600558 23299 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35701
I20260927 05:11:52.600579 23365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35701 every 8 connection(s)
I20260927 05:11:52.602118 23299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/data/info.pb
I20260927 05:11:52.603258 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 23299
I20260927 05:11:52.603478 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-1/wal/instance
I20260927 05:11:52.604087 388 external_mini_cluster.cc:1428] Setting key 50cbf23c127ea0f4222103bc359e78d2
I20260927 05:11:52.607719 23366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:52.618501 23366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 27 05:11:52 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:52.644447 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.621439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.0.97.62:34707)
0927 05:11:52.621998 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:52.622003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:52.623116 (+ 1113us) server_negotiation.cc:407] Connection header received
0927 05:11:52.625156 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:52.625163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:52.625258 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:52.625382 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:52.626973 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.627693 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.629401 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.629815 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.629955 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:52.636701 (+ 6746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:52.636710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:52.636719 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:52.636787 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:52.640566 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:52.641544 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:52.641553 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:52.641559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:52.641716 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:52.642180 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:52.642188 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:52.642194 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:52.642646 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:52.642818 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:52.643629 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 05:11:52.643903 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":329,"threads_started":1}
I20260927 05:11:52.645206 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.622135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34707 (local address 127.0.0.1:56782)
0927 05:11:52.622936 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.622984 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:52.624853 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:52.625637 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:52.625646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:52.626184 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:52.626756 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:52.626773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.627864 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.627868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:52.629237 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:52.629250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.630051 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.630058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:52.630309 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:52.631052 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:52.631078 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:52.636385 (+ 5307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:52.640811 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:52.640820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:52.640835 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:52.641374 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:52.641847 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:52.641850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:52.641852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:52.642048 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:52.642832 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:52.642839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:52.643218 (+ 379us) client_negotiation.cc:769] Sending connection context
0927 05:11:52.643463 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 05:11:52.643928 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":357,"threads_started":1}
I20260927 05:11:52.647382 23366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:11:52 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:11:52.662410 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.648267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41378)
0927 05:11:52.648471 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.648489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:52.648607 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:52.649616 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:52.649620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:52.650063 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:52.650198 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:52.650206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.651615 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.651621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:52.652537 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:52.652549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.653366 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.653372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:52.653563 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:52.654168 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:52.654185 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:52.656457 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:52.659739 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:52.659750 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:52.659755 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:52.660170 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:52.660539 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:52.660543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:52.660549 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:52.660676 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:52.661381 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:52.661385 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:52.661548 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 05:11:52.661686 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 05:11:52.661991 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 05:11:52.662505 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.648445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.0.97.61:35701)
0927 05:11:52.649202 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:52.649209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:52.649234 (+ 25us) server_negotiation.cc:407] Connection header received
0927 05:11:52.649278 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:52.649284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:52.649469 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:52.649628 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:52.650380 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.651461 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.652673 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.653221 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.653359 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:52.656668 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:52.656679 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:52.656694 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:52.656726 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:52.659594 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:52.660297 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:52.660312 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:52.660314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:52.660424 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:52.660794 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:52.660798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:52.660801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:52.661242 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:52.661468 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:52.661852 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 05:11:52.662115 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":601,"thread_start_us":457,"threads_started":1}
I20260927 05:11:52.664907 23366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:11:52.666036 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.665576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:39918)
0927 05:11:52.665777 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.665890 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 05:11:52.666309 23318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
I20260927 05:11:52.668056 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.667631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:39932)
0927 05:11:52.667768 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.667844 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 05:11:52.668298 23366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43235: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260927 05:11:52.672087 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
/tmp/dist-test-taskEQLg3e/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43235
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34707,127.0.97.61:35701,127.0.97.60:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 05:11:52.717641 23366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } attempt: 1
I20260927 05:11:52.718904 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.718386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:39948)
0927 05:11:52.718622 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.718719 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 05:11:52.720068 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.719652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:39958)
0927 05:11:52.719817 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.719888 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 05:11:52.720340 23366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43235: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
I20260927 05:11:52.749750 23292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } attempt: 5
Sep 27 05:11:52 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:11:52.764984 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.750611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41390)
0927 05:11:52.750874 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.750900 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:52.751045 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:52.751452 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:52.751456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:52.751847 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:52.752140 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:52.752151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.753820 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.753825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:52.754755 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:52.754779 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.755692 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.755699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:52.755928 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:52.756681 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:52.756701 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:52.759107 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:52.762387 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:52.762392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:52.762395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:52.762808 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:52.763200 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:52.763205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:52.763208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:52.763356 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:52.763972 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:52.763977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:52.764133 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 05:11:52.764307 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 05:11:52.764615 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 05:11:52.764976 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.750794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.0.97.61:35701)
0927 05:11:52.750999 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:52.751008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:52.751036 (+ 28us) server_negotiation.cc:407] Connection header received
0927 05:11:52.751172 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:52.751178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:52.751261 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:52.751376 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:52.752309 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.753607 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.754977 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:52.755508 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:52.755673 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:52.759323 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:52.759328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:52.759331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:52.759366 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:52.762221 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:52.762935 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:52.762943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:52.762950 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:52.763080 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:52.763479 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:52.763485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:52.763489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:52.763796 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:52.764000 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:52.764428 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 05:11:52.764644 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 05:11:52.766395 23292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:11:52.767522 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.767063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:39968)
0927 05:11:52.767231 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.767315 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 05:11:52.768563 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.768139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:39980)
0927 05:11:52.768302 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.768393 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 05:11:52.768850 23292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43235: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
I20260927 05:11:52.788615 23366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } attempt: 2
I20260927 05:11:52.789930 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.789454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:39984)
0927 05:11:52.789663 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.789758 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 05:11:52.790990 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.790611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:39992)
0927 05:11:52.790760 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.790844 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 05:11:52.791239 23366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43235: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
I20260927 05:11:52.820102 23292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } attempt: 1
I20260927 05:11:52.821475 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.820861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40002)
0927 05:11:52.821105 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.821245 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 05:11:52.822650 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.822201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40018)
0927 05:11:52.822395 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.822496 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 05:11:52.822923 23292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43235: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
I20260927 05:11:52.882494 23366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } attempt: 3
I20260927 05:11:52.883760 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.883219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40026)
0927 05:11:52.883462 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.883568 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 05:11:52.885032 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.884502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40040)
0927 05:11:52.884799 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.884894 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260927 05:11:52.885344 23366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43235: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
I20260927 05:11:52.891156 23292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } attempt: 2
I20260927 05:11:52.892228 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.891708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40050)
0927 05:11:52.891909 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.892043 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 05:11:52.893179 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:52.892826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40056)
0927 05:11:52.892967 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:52.893057 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 05:11:52.893384 23292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43235: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
I20260927 05:11:52.999617 23292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } attempt: 3
I20260927 05:11:53.000845 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.000291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40066)
0927 05:11:53.000524 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:53.000640 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 05:11:53.002012 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.001561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40070)
0927 05:11:53.001797 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:53.001878 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 05:11:53.002254 23292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43235: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
W20260927 05:11:53.014182 23374 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 05:11:53.014541 23374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 05:11:53.014684 23374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 05:11:53.014808 23374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 05:11:53.022482 23374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 05:11:53.022588 23374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 05:11:53.022687 23374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 05:11:53.022775 23374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 05:11:53.022883 23374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 05:11:53.028585 23366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } attempt: 4
I20260927 05:11:53.029789 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.029248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40084)
0927 05:11:53.029498 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:53.029604 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 05:11:53.030821 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.030475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40086)
0927 05:11:53.030619 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:53.030698 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 05:11:53.031056 23366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:43235: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43235: connect: Connection refused (error 111)
I20260927 05:11:53.033890 23374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34707,127.0.97.61:35701,127.0.97.60:43235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 04:08:07 UTC on ccycloud.quasar-zxfvcz.root.comops.site
build id 4675662
ASAN enabled
I20260927 05:11:53.035251 23374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 05:11:53.036689 23374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 05:11:53.049993 23380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:53.050290 23379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 05:11:53.050957 23382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 05:11:53.051322 23374 server_base.cc:1034] running on GCE node
I20260927 05:11:53.052259 23374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 05:11:53.053879 23374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 05:11:53.055143 23374 hybrid_clock.cc:648] HybridClock initialized: now 1790485913055056 us; error 53 us; skew 500 ppm
I20260927 05:11:53.055541 23374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 05:11:53 dist-test-slave-hgs0 krb5kdc[23212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 05:11:53.058866 23374 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260927 05:11:53.060452 23374 webserver.cc:460] Webserver started at http://127.0.97.60:43437/ using document root <none> and password file <none>
I20260927 05:11:53.061230 23374 fs_manager.cc:359] Metadata directory not provided
I20260927 05:11:53.061367 23374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 05:11:53.061719 23374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 05:11:53.064980 23374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/data/instance:
uuid: "24a921b923b043c58bca158460d1b93c"
format_stamp: "Formatted at 2026-09-27 05:11:53 on dist-test-slave-hgs0"
server_key: "7a2cf8d623df9b3e990b7a57e9b68278"
server_key_iv: "f452f6b9f7c2cdd56ca9c5e383e9955b"
server_key_version: "encryptionkey@0"
I20260927 05:11:53.065833 23374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/wal/instance:
uuid: "24a921b923b043c58bca158460d1b93c"
format_stamp: "Formatted at 2026-09-27 05:11:53 on dist-test-slave-hgs0"
server_key: "7a2cf8d623df9b3e990b7a57e9b68278"
server_key_iv: "f452f6b9f7c2cdd56ca9c5e383e9955b"
server_key_version: "encryptionkey@0"
I20260927 05:11:53.071254 23374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 05:11:53.075124 23389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 05:11:53.076843 23374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 05:11:53.077059 23374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/wal
uuid: "24a921b923b043c58bca158460d1b93c"
format_stamp: "Formatted at 2026-09-27 05:11:53 on dist-test-slave-hgs0"
server_key: "7a2cf8d623df9b3e990b7a57e9b68278"
server_key_iv: "f452f6b9f7c2cdd56ca9c5e383e9955b"
server_key_version: "encryptionkey@0"
I20260927 05:11:53.077307 23374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 05:11:53.095072 23374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 05:11:53.099413 23374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 05:11:53.099713 23374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 05:11:53.119212 23374 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43235
I20260927 05:11:53.119234 23440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43235 every 8 connection(s)
I20260927 05:11:53.120709 23374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/data/info.pb
I20260927 05:11:53.124630 23441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 05:11:53.129465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu as pid 23374
I20260927 05:11:53.129720 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskEQLg3e/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790485535376807-388-0/minicluster-data/master-2/wal/instance
I20260927 05:11:53.130172 388 external_mini_cluster.cc:1428] Setting key 5006d2fc09f5b114b321507dc39ca852
I20260927 05:11:53.132144 23441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:11:53 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 27 05:11:53 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:11:53.150755 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.135465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.0.97.62:34707)
0927 05:11:53.135658 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.135664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.136433 (+ 769us) server_negotiation.cc:407] Connection header received
0927 05:11:53.136703 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.136707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.136804 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.136916 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.138183 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.138866 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.140436 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.140867 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.141003 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.144771 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.144776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.144779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.144813 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.147737 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.148522 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.148526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.148528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.148647 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.149064 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.149069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.149077 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.149335 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.149489 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.150189 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.150423 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 05:11:53.151829 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.132145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.0.97.60:43235)
0927 05:11:53.133029 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.133046 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.133097 (+ 51us) server_negotiation.cc:407] Connection header received
0927 05:11:53.135375 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.135398 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.135875 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.136330 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.137780 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.138978 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.140378 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.140905 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.141120 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.145425 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.145434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.145447 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.145477 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.148245 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.148926 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.148933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.148935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.149083 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.149467 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.149472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.149474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.149848 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.150172 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.150469 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.150720 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":343,"threads_started":1}
I20260927 05:11:53.151866 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.135464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34707 (local address 127.0.0.1:56798)
0927 05:11:53.136302 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:53.136350 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:53.136549 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:53.136985 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:53.136992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:53.137597 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:53.138014 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:53.138027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.139028 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.139038 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:53.140285 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:53.140297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.141116 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.141122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:53.141279 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.142048 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:53.142077 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:53.144518 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:53.147871 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:53.147885 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:53.147900 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:53.148388 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:53.148754 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:53.148760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:53.148764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:53.148936 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:53.149464 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:53.149472 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:53.149831 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 05:11:53.150085 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 05:11:53.150584 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":427,"threads_started":1}
I20260927 05:11:53.153584 23441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 05:11:53 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 27 05:11:53 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260927 05:11:53.167603 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.154304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35701 (local address 127.0.0.1:41402)
0927 05:11:53.154455 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:53.154475 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:53.154595 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:53.154906 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:53.154909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:53.155359 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:53.155547 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:53.155603 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.156745 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.156751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:53.157638 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:53.157649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.158336 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.158342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:53.158532 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.159130 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:53.159153 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:53.161357 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:53.165161 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:53.165165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:53.165167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:53.165590 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:53.166016 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:53.166019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:53.166022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:53.166161 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:53.166752 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:53.166756 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:53.166878 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 05:11:53.167005 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 05:11:53.167213 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 05:11:53.167618 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.154383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.0.97.61:35701)
0927 05:11:53.154582 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.154588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.154609 (+ 21us) server_negotiation.cc:407] Connection header received
0927 05:11:53.154698 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.154703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.154780 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.154888 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.155763 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.156593 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.157782 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.158188 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.158317 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.161548 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.161554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.161557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.161588 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.165010 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.165716 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.165723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.165730 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.165895 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.166268 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.166273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.166276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.166634 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.166790 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.167128 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.167345 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:11:53.168741 23441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } has no permanent_uuid. Determining permanent_uuid...
I20260927 05:11:53.170354 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.155344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.0.97.62:34707)
0927 05:11:53.155511 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.155517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.155535 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:11:53.155637 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.155644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.155732 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.155844 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.156964 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.157653 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.158837 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.159468 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.159606 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.163996 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.164003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.164009 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.164062 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.167212 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.168244 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.168250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.168253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.168402 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.168778 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.168783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.168787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.169104 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.169424 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.169781 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.169996 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 05:11:53 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:11:53.179459 23292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } attempt: 4
Sep 27 05:11:53 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260927 05:11:53.183164 23444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.169637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.0.97.60:43235)
0927 05:11:53.169793 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.169798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.169815 (+ 17us) server_negotiation.cc:407] Connection header received
0927 05:11:53.169919 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.169924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.169998 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.170092 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.171331 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.172170 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.173415 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.173916 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.174011 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.177198 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.177202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.177205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.177234 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.180089 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.180977 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.180981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.180984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.181117 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.181537 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.181543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.181547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.181912 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.182071 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.182514 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.182855 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":61}
I20260927 05:11:53.183163 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.169402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40096)
0927 05:11:53.169682 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:53.169700 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:53.169789 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:53.170134 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:53.170141 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:53.170611 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:53.170738 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:53.170746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.172299 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.172307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:53.173276 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:53.173287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.174069 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.174076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:53.174288 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.174884 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:53.174900 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:53.177019 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:53.180254 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:53.180266 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:53.180272 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:53.180849 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:53.181254 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:53.181258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:53.181262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:53.181420 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:53.182113 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:53.182116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:53.182249 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 05:11:53.182411 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 05:11:53.182717 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":3}
I20260927 05:11:53.187477 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.172069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.0.97.61:35701)
0927 05:11:53.172208 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.172212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.172252 (+ 40us) server_negotiation.cc:407] Connection header received
0927 05:11:53.172345 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.172352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.172421 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.172515 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.173651 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.174602 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.175751 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.176287 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.176381 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.180947 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.180955 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.180959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.180999 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.184441 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.185066 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.185072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.185075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.185253 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.185698 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.185703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.185707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.186053 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.186323 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.186895 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.187122 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 05:11:53 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260927 05:11:53.190073 23441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c: Bootstrap starting.
I20260927 05:11:53.192981 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 05:11:53.194446 23441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:53.195688 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.180078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40110)
0927 05:11:53.180569 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:53.180592 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:53.180727 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:53.181215 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:53.181222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:53.181660 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:53.181863 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:53.181878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.183740 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.183746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:53.185197 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:53.185219 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.186033 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.186039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:53.186244 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.186841 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:53.186863 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:53.189711 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:53.193231 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:53.193236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:53.193240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:53.193605 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:53.193984 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:53.193991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:53.193995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:53.194150 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:53.194842 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:53.194852 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:53.195010 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 05:11:53.195149 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 05:11:53.195402 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355}
I20260927 05:11:53.195829 23441 log.cc:824] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:53.196761 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.180136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.0.97.60:43235)
0927 05:11:53.180935 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.180940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.180967 (+ 27us) server_negotiation.cc:407] Connection header received
0927 05:11:53.181003 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.181006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.181057 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.181179 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.182121 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.183592 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.185342 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.185844 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.186547 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.189933 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.189938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.189941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.189994 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.193118 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.193731 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.193736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.193740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.193873 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.194266 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.194271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.194274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.194628 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.194767 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.196059 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.196269 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":668,"thread_start_us":685,"threads_started":1}
I20260927 05:11:53.199514 23441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c: No bootstrap required, opened a new log
I20260927 05:11:53.195990 23267 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6de2d1b3e6484b77b37232953392b65e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56814
I20260927 05:11:53.201238 23292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e: Bootstrap starting.
I20260927 05:11:53.205210 23292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:53.206530 23292 log.cc:824] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:53.207211 23441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } }
I20260927 05:11:53.207677 23441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a921b923b043c58bca158460d1b93c, State: Initialized, Role: FOLLOWER
I20260927 05:11:53.208200 23441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } }
I20260927 05:11:53.209112 23449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } } }
I20260927 05:11:53.209582 23449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:53.209962 23292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e: No bootstrap required, opened a new log
I20260927 05:11:53.210366 23441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:53.217309 23292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } }
I20260927 05:11:53.217736 23292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de2d1b3e6484b77b37232953392b65e, State: Initialized, Role: FOLLOWER
I20260927 05:11:53.218250 23292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } }
I20260927 05:11:53.219240 23453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } } }
I20260927 05:11:53.219691 23453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:53.220659 23292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:53.220788 23267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 05:11:53.221042 23267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:53.225406 23267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } }
I20260927 05:11:53.228560 23416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6de2d1b3e6484b77b37232953392b65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "24a921b923b043c58bca158460d1b93c"
I20260927 05:11:53.229190 23416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:53.233919 23416 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6de2d1b3e6484b77b37232953392b65e in term 1.
I20260927 05:11:53.234465 23267 leader_election.cc:288] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [CANDIDATE]: Term 1 election: Requested vote from peers 44717074f0294bb4b465fb549300697d (127.0.97.61:35701), 24a921b923b043c58bca158460d1b93c (127.0.97.60:43235)
I20260927 05:11:53.234912 23244 leader_election.cc:302] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a921b923b043c58bca158460d1b93c, 6de2d1b3e6484b77b37232953392b65e; no voters:
I20260927 05:11:53.235581 23453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 05:11:53.250273 23453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [term 1 LEADER]: Becoming Leader. State: Replica: 6de2d1b3e6484b77b37232953392b65e, State: Running, Role: LEADER
I20260927 05:11:53.252193 23453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } }
W20260927 05:11:53.274509 23246 tablet.cc:2369] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 05:11:53.283314 23476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 05:11:53.283471 23476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 05:11:53.283800 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.252770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.0.97.61:35701)
0927 05:11:53.253092 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.253102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.254960 (+ 1858us) server_negotiation.cc:407] Connection header received
0927 05:11:53.256019 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.256024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.256102 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.256172 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.260781 (+ 4609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.262669 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.266375 (+ 3706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.267107 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.267222 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.274431 (+ 7209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.274441 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.274447 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.274511 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.278319 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.280082 (+ 1763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.280089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.280097 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.280287 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.281240 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.281246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.281248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.281685 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.281974 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.283094 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.283390 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 05:11:53.292533 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.253619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.0.97.62:34707)
0927 05:11:53.253825 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.253833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.254993 (+ 1160us) server_negotiation.cc:407] Connection header received
0927 05:11:53.255735 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.255744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.255853 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.255964 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.264629 (+ 8665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.266108 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.272115 (+ 6007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.273208 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.273354 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.283680 (+ 10326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.283687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.283690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.283729 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.288951 (+ 5222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.289889 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.289895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.289899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.290055 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.290558 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.290564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.290567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.290973 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.291245 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.291821 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.292100 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 05:11:53.296237 23463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6de2d1b3e6484b77b37232953392b65e. Latest consensus state: current_term: 1 leader_uuid: "6de2d1b3e6484b77b37232953392b65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } } }
I20260927 05:11:53.297441 23463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:53.308135 23480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 05:11:53.309551 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.255042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.0.97.60:43235)
0927 05:11:53.255302 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.255307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.260760 (+ 5453us) server_negotiation.cc:407] Connection header received
0927 05:11:53.261989 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.261994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.262091 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.262175 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.268669 (+ 6494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.270157 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.279777 (+ 9620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.280601 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.280705 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.285116 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.285122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.285127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.285173 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.292287 (+ 7114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.294141 (+ 1854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.294147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.294151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.294301 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.304772 (+ 10471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.304780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.304784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.305392 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.308524 (+ 3132us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.308734 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.309029 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:11:53.264953 23374 master_runner.cc:186] Error getting master registration for 127.0.97.62:34707: OK, instance_id { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" instance_seqno: 1790485912034805 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:11:53.310503 23374 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:11:53.325691 23480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 05:11:53.330260 23366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } attempt: 5
I20260927 05:11:53.334057 23484 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Sep 27 05:11:53 dist-test-slave-hgs0 krb5kdc[23212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790485912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260927 05:11:53.352039 23395 tablet.cc:2369] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 05:11:53.341864 23225 master_runner.cc:186] Error getting master registration for 127.0.97.61:35701: OK, instance_id { permanent_uuid: "44717074f0294bb4b465fb549300697d" instance_seqno: 1790485912594541 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 05:11:53.352906 23225 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 05:11:53.358445 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.330948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43235 (local address 127.0.0.1:40132)
0927 05:11:53.331319 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 05:11:53.331340 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 05:11:53.331454 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 05:11:53.332391 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 05:11:53.332396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 05:11:53.332950 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 05:11:53.333162 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:53.333175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.339767 (+ 6592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.339773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:53.341331 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 05:11:53.341350 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.343284 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.343291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 05:11:53.343526 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.344292 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 05:11:53.344319 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 05:11:53.347312 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 05:11:53.355105 (+ 7793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:53.355111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:53.355115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:53.355716 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:53.356239 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 05:11:53.356245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 05:11:53.356250 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 05:11:53.356427 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 05:11:53.357234 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 05:11:53.357242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 05:11:53.357432 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 05:11:53.357618 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 05:11:53.357984 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260927 05:11:53.359479 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.331686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40132 (local address 127.0.97.60:43235)
0927 05:11:53.331867 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.331873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.331892 (+ 19us) server_negotiation.cc:407] Connection header received
0927 05:11:53.331987 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.331993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.332072 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.332178 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.333343 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.334568 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.342091 (+ 7523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.342876 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.343021 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.347577 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 05:11:53.347585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 05:11:53.347590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 05:11:53.347662 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 05:11:53.354892 (+ 7230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.355876 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.355882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.355886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.356089 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 05:11:53.356568 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 05:11:53.356573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 05:11:53.356576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 05:11:53.356983 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 05:11:53.357152 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.358792 (+ 1640us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.359047 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 05:11:53.363293 23416 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [term 1 FOLLOWER]: Refusing update from remote peer 6de2d1b3e6484b77b37232953392b65e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:11:53.365397 23453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:11:53.365706 23366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d: Bootstrap starting.
I20260927 05:11:53.371784 23366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d: Neither blocks nor log segments found. Creating new log.
I20260927 05:11:53.373237 23366 log.cc:824] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d: Log is configured to *not* fsync() on all Append() calls
I20260927 05:11:53.386363 23366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d: No bootstrap required, opened a new log
I20260927 05:11:53.393707 23449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6de2d1b3e6484b77b37232953392b65e. Latest consensus state: current_term: 1 leader_uuid: "6de2d1b3e6484b77b37232953392b65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } } }
I20260927 05:11:53.394131 23449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:53.398847 23366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } }
I20260927 05:11:53.399374 23366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44717074f0294bb4b465fb549300697d, State: Initialized, Role: FOLLOWER
I20260927 05:11:53.400072 23366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } }
I20260927 05:11:53.404520 23449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6de2d1b3e6484b77b37232953392b65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } } }
I20260927 05:11:53.404829 23449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:53.411368 23341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6de2d1b3e6484b77b37232953392b65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "44717074f0294bb4b465fb549300697d"
I20260927 05:11:53.408496 23491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } } }
I20260927 05:11:53.411896 23491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:53.411244 23366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 05:11:53.411993 23341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [term 0 FOLLOWER]: Advancing to term 1
I20260927 05:11:53.416908 23341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6de2d1b3e6484b77b37232953392b65e in term 1.
I20260927 05:11:53.417979 23340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [term 1 FOLLOWER]: Refusing update from remote peer 6de2d1b3e6484b77b37232953392b65e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 05:11:53.420739 23453 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 05:11:53.421052 23480 catalog_manager.cc:1347] Generated new cluster ID: ad69f0b90dcf43e4a0e0810293610022
I20260927 05:11:53.421175 23480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 05:11:53.426923 23463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6de2d1b3e6484b77b37232953392b65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } } }
I20260927 05:11:53.427335 23463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:53.454399 23480 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 05:11:53.456843 23480 catalog_manager.cc:1504] Loading token signing keys...
I20260927 05:11:53.463587 23494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6de2d1b3e6484b77b37232953392b65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } } }
I20260927 05:11:53.464151 23494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e [sys.catalog]: This master's current role is: LEADER
I20260927 05:11:53.464720 23491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6de2d1b3e6484b77b37232953392b65e. Latest consensus state: current_term: 1 leader_uuid: "6de2d1b3e6484b77b37232953392b65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } } }
I20260927 05:11:53.465070 23491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:53.464691 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6de2d1b3e6484b77b37232953392b65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6de2d1b3e6484b77b37232953392b65e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34707 } } peers { permanent_uuid: "44717074f0294bb4b465fb549300697d" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35701 } } peers { permanent_uuid: "24a921b923b043c58bca158460d1b93c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43235 } } }
I20260927 05:11:53.465188 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 05:11:53.489619 23480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6de2d1b3e6484b77b37232953392b65e: Generated new TSK 0
I20260927 05:11:53.494010 23480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 05:11:53.521611 23508 catalog_manager.cc:1259] Loaded cluster ID: ad69f0b90dcf43e4a0e0810293610022
I20260927 05:11:53.521828 23508 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d: loading cluster ID for follower catalog manager: success
I20260927 05:11:53.526144 23508 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d: acquiring CA information for follower catalog manager: success
I20260927 05:11:53.538998 23508 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44717074f0294bb4b465fb549300697d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 05:11:53.626751 23372 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:53.617497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.0.97.61:35701)
0927 05:11:53.617728 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.617736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.623290 (+ 5554us) server_negotiation.cc:407] Connection header received
0927 05:11:53.625117 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.625124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.625213 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.625291 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.626468 (+ 1177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41444: BlockingRecv error: recv got EOF from 127.0.0.1:41444 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 05:11:53.632694 23446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:53.623856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.0.97.60:43235)
0927 05:11:53.624063 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.624084 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.629442 (+ 5358us) server_negotiation.cc:407] Connection header received
0927 05:11:53.631043 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.631049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.631140 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.631214 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.632404 (+ 1190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40140: BlockingRecv error: recv got EOF from 127.0.0.1:40140 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 05:11:53.633549 23369 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:53.618562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.0.97.62:34707)
0927 05:11:53.618822 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.618828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.627902 (+ 9074us) server_negotiation.cc:407] Connection header received
0927 05:11:53.628832 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.628839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.628922 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.628997 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.633288 (+ 4291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56840: BlockingRecv error: recv got EOF from 127.0.0.1:56840 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 05:11:53.682777 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.667114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.0.97.61:35701)
0927 05:11:53.667323 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.667330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.674688 (+ 7358us) server_negotiation.cc:407] Connection header received
0927 05:11:53.675193 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.675202 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.675431 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.675588 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:53.676251 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.677464 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.679449 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.680167 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.680275 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.681131 (+ 856us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:53.681891 (+ 760us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:53.682020 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.682323 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.682426 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 05:11:53.726423 23446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:53.717032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.0.97.60:43235)
0927 05:11:53.717245 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.717251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.724231 (+ 6980us) server_negotiation.cc:407] Connection header received
0927 05:11:53.724816 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.724824 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.724964 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.725055 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.726168 (+ 1113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40150: BlockingRecv error: recv got EOF from 127.0.0.1:40150 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 05:11:53.772662 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.757053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.0.97.61:35701)
0927 05:11:53.757286 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.757297 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.765649 (+ 8352us) server_negotiation.cc:407] Connection header received
0927 05:11:53.766180 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.766187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.766280 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.766344 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:53.767022 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.768203 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.770324 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.770947 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.771068 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.771708 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:53.771879 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:53.771977 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.772216 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.772310 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 05:11:53.816295 23446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:53.807008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.0.97.60:43235)
0927 05:11:53.807191 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.807197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.814210 (+ 7013us) server_negotiation.cc:407] Connection header received
0927 05:11:53.814762 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.814769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.814879 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.814969 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.816052 (+ 1083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40164: BlockingRecv error: recv got EOF from 127.0.0.1:40164 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 05:11:53.870224 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.855457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.0.97.61:35701)
0927 05:11:53.855670 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.855676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.863160 (+ 7484us) server_negotiation.cc:407] Connection header received
0927 05:11:53.863702 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.863711 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.863840 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.863908 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:53.864632 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.865821 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.867839 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.868541 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.868652 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.869285 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:53.869462 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:53.869527 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.869779 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.869870 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 05:11:53.922991 23446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:53.905374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.0.97.60:43235)
0927 05:11:53.905606 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.905615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.921110 (+ 15495us) server_negotiation.cc:407] Connection header received
0927 05:11:53.921382 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.921389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.921489 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.921612 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:53.922726 (+ 1114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40168: BlockingRecv error: recv got EOF from 127.0.0.1:40168 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:11:53.984679 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:53.969386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.0.97.61:35701)
0927 05:11:53.969592 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:53.969598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:53.977090 (+ 7492us) server_negotiation.cc:407] Connection header received
0927 05:11:53.977668 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:53.977676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:53.977801 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:53.977877 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:53.978577 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.979778 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.982107 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:53.982849 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:53.982968 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:53.983738 (+ 770us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:53.983897 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:53.984001 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:53.984256 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 05:11:53.984348 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 05:11:54.018380 23446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:54.009978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.0.97.60:43235)
0927 05:11:54.010284 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.010291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.016640 (+ 6349us) server_negotiation.cc:407] Connection header received
0927 05:11:54.016897 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.016905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.017029 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.017156 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:54.018134 (+ 978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40172: BlockingRecv error: recv got EOF from 127.0.0.1:40172 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 05:11:54.038168 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:54.030221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.0.97.61:35701)
0927 05:11:54.030414 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.030422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.030925 (+ 503us) server_negotiation.cc:407] Connection header received
0927 05:11:54.031242 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.031248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.031356 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.031429 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:54.032198 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.033426 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.035555 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.036368 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.036482 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:54.037207 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:54.037361 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:54.037439 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:54.037699 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 05:11:54.037799 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 05:11:54.078306 23446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:54.073319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.0.97.60:43235)
0927 05:11:54.073537 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.073545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.076622 (+ 3077us) server_negotiation.cc:407] Connection header received
0927 05:11:54.076840 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.076846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.076947 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.077074 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:54.078030 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40182: BlockingRecv error: recv got EOF from 127.0.0.1:40182 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 05:11:54.105268 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:54.098892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.0.97.61:35701)
0927 05:11:54.099074 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.099080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.099608 (+ 528us) server_negotiation.cc:407] Connection header received
0927 05:11:54.099824 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.099830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.099925 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.100063 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:54.100552 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.101734 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.103100 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.103826 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.103988 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:54.104492 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:54.104635 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:54.104724 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:54.104857 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 05:11:54.104951 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 05:11:54.115483 23446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:54.112999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.0.97.60:43235)
0927 05:11:54.113288 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.113300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.113788 (+ 488us) server_negotiation.cc:407] Connection header received
0927 05:11:54.114022 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.114026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.114118 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.114229 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:54.115215 (+ 986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40192: BlockingRecv error: recv got EOF from 127.0.0.1:40192 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 05:11:54.157917 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:54.151854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.0.97.61:35701)
0927 05:11:54.152098 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.152102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.152578 (+ 476us) server_negotiation.cc:407] Connection header received
0927 05:11:54.152873 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.152879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.152972 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.153150 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:54.153548 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.154357 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.155849 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.156523 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.156648 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:54.157160 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:54.157307 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:54.157389 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:54.157577 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 05:11:54.157655 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 05:11:54.166141 23446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:54.164268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.0.97.60:43235)
0927 05:11:54.164448 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.164453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.164809 (+ 356us) server_negotiation.cc:407] Connection header received
0927 05:11:54.164942 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.164947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.165025 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.165125 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:54.165916 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40206: BlockingRecv error: recv got EOF from 127.0.0.1:40206 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 05:11:54.240109 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:54.234335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.0.97.61:35701)
0927 05:11:54.234608 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.234626 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.234868 (+ 242us) server_negotiation.cc:407] Connection header received
0927 05:11:54.235002 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.235007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.235097 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.235188 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:54.235669 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.236585 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.238078 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.238509 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.238618 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:54.239116 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:54.239266 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:54.239369 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:54.239611 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 05:11:54.239688 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260927 05:11:54.249697 23446 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 05:11:54.247414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.0.97.60:43235)
0927 05:11:54.247666 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.247670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.248162 (+ 492us) server_negotiation.cc:407] Connection header received
0927 05:11:54.248295 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.248300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.248388 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.248546 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 05:11:54.249460 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40220: BlockingRecv error: recv got EOF from 127.0.0.1:40220 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 05:11:54.285517 23476 catalog_manager.cc:1259] Loaded cluster ID: ad69f0b90dcf43e4a0e0810293610022
I20260927 05:11:54.285631 23476 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c: loading cluster ID for follower catalog manager: success
I20260927 05:11:54.288331 23476 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c: acquiring CA information for follower catalog manager: success
I20260927 05:11:54.290447 23476 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 24a921b923b043c58bca158460d1b93c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 05:11:54.387856 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:54.382101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.0.97.61:35701)
0927 05:11:54.382300 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.382307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.382925 (+ 618us) server_negotiation.cc:407] Connection header received
0927 05:11:54.383073 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.383078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.383159 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.383248 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:54.383628 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.384531 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.386065 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.386519 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.386635 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:54.387074 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:54.387194 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:54.387266 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:54.387408 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 05:11:54.387480 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 05:11:54.400976 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:54.394523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.0.97.60:43235)
0927 05:11:54.394777 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.394785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.395026 (+ 241us) server_negotiation.cc:407] Connection header received
0927 05:11:54.395193 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.395198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.395364 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.395476 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:54.395928 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.396792 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.398280 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.398820 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.398944 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:54.399527 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:54.400276 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:54.400478 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:54.400559 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 05:11:54.400656 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 05:11:54.416625 23446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:54.409016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.0.97.60:43235)
0927 05:11:54.409248 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.409254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.410394 (+ 1140us) server_negotiation.cc:407] Connection header received
0927 05:11:54.410649 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.410655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.410744 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.410849 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:54.411845 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.412758 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.414668 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.415210 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.415364 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:54.415812 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:54.415992 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:54.416110 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:54.416257 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 05:11:54.416341 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 05:11:54.417510 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:54.408225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.0.97.61:35701)
0927 05:11:54.408423 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.408433 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.410115 (+ 1682us) server_negotiation.cc:407] Connection header received
0927 05:11:54.410449 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.410456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.410570 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.410636 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:54.411868 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.412803 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.414911 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.415594 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.415717 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:54.416404 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:54.416560 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:54.416649 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:54.417169 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 05:11:54.417255 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 05:11:54.419759 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 05:11:54.408033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.0.97.62:34707)
0927 05:11:54.410519 (+ 2486us) server_negotiation.cc:206] Beginning negotiation
0927 05:11:54.410523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 05:11:54.410541 (+ 18us) server_negotiation.cc:407] Connection header received
0927 05:11:54.410686 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 05:11:54.410691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 05:11:54.410919 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 05:11:54.411141 (+ 222us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 05:11:54.412963 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.414124 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.416912 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 05:11:54.417436 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 05:11:54.417581 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 05:11:54.418639 (+ 1058us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 05:11:54.419070 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 05:11:54.419221 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 05:11:54.419362 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 05:11:54.419468 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2377,"thread_start_us":2279,"threads_started":1}
I20260927 05:11:54.430191 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 23225
I20260927 05:11:54.431465 23287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:54.555168 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 23225
W20260927 05:11:54.577358 23391 connection.cc:535] server connection from 127.0.0.1:40110 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 05:11:54.577744 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 23299
I20260927 05:11:54.578966 23361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:54.695070 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 23299
I20260927 05:11:54.717018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu pid 23374
I20260927 05:11:54.718300 23436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 05:11:54.834056 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskEQLg3e/build/asan/bin/kudu with pid 23374
2026-09-27T05:11:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4668 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9193 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (379415 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST